Principles Library / Earned Agent Trust

Earned Agent Trust

Agent Trust Must Be Continuously Earned, Not Historically Assumed

Act

Showing all 5 practices. Answer the Diagnose questions and the most relevant come first.

Set trust criteria before deployment

Before deployment, agree what observable evidence, reviewed on what cadence, keeps confidence in the agent, and what you scale back first when it falls short.

Start by asking your team: What would have to be true for you to reduce trust in this agent?

Avoid · You’ll know it’s working when

Avoid

    You’ll know it’s working when

    The team can name the evidence that would make it look harder at the agent, and what it would scale back first.

    Keep monitoring and evaluation as separate jobs

    Run automated, continuous monitoring and human, periodic, outcome-focused evaluation side by side. You need both. Neither one stands in for the other.

    Start by asking your team: When did you last check agent outputs against today's organizational needs rather than past outputs?

    Avoid · You’ll know it’s working when

    Avoid

    • Treating monitoring (watching outputs) as if it were evaluation (judging fitness for purpose)
    • Reading shorter review time as mature trust when it's eroded oversight

    You’ll know it’s working when

    Sign-off means someone actually weighed the outputs, and review time only shrinks with a written reason.

    Trigger a review when things change

    Start a structured review whenever there's a significant organizational change, a context shift or a drop in output quality.

    Start by asking your team: Has anything changed in the organization since deployment that might affect whether outputs are still fit for purpose?

    Avoid · You’ll know it’s working when

    Avoid

    • Assuming an agent that did well in one context will do well in a changed one without re-evaluation
    • Believing vendor model updates keep the agent reliable without your own re-evaluation

    You’ll know it’s working when

    Each organizational change or vendor model update starts a review of the agent's outputs and policies.

    Keep current evidence behind your confidence

    Make sure the team can say, at any time, what current evidence supports its confidence in the agent.

    Start by asking your team: What evidence from the last two weeks tells you this agent is still performing correctly?

    Avoid · You’ll know it’s working when

    Avoid

      You’ll know it’s working when

      Asked about reliability, the team points to outputs it checked recently.

      Treat quiet stretches as a monitoring check

      When the agent has run a long time without incidents, take that as a cue to check whether monitoring is sufficient.

      Start by asking your team: Which monitoring metrics would catch gradual degradation versus sudden failure?

      Avoid · You’ll know it’s working when

      Avoid

      • Treating no incidents as proof the agent is reliable

      You’ll know it’s working when

      Internal review catches agent failures before users or downstream systems do.

      Overview

      Principle intent

      Trust in an agent system must be grounded in current, observed behavior, not in historical performance.

      Read the full intent

      An agent that performed reliably in the past provides no guarantee of reliable performance today. As context shifts, configurations age, and novel situations arise, trust based on history becomes a governance liability.

      When it holds: what it does for you

      Failures surface early

      A drop in output quality starts a review before users find the problem.

      Confidence you can explain

      The team can say what current evidence backs its trust in the agent.

      Steady, calibrated oversight

      One bad failure doesn't send everyone scrambling back to manual work.

      When it is ignored: from early signs to a lasting condition

      Early warning signs · Visible now, cheap to fix

      • Agent reviews keep getting shorter, and nobody has written down why.
      • Asked if the agent is reliable, the team points to time since the last incident.
      • Nobody can say what evidence would make them look harder at the agent.

      What happens over time · Accumulates, costly to reverse

      • Reviews thinned by past success miss early signals of serious agent failures.
      • Post-incident reviews find the warning signs were on record, and nobody looked for them.
      • After a big failure, trust collapses into manual processes that hurt more than calibrated oversight.

      The condition it becomes

      Oversight Erosion · Primary

      The review process still runs. It stopped meaning anything.

      How this principle feeds it

      review keeps running but stops catching failures, which is the condition's defining state.

      Governance Drag · Secondary

      Work is done. Approvals are not.

      How this principle feeds it

      manual checks on every agent output add delay well beyond the risk they remove.

      Intent Drift · Primary

      The team built exactly what was asked. It is no longer what is needed.

      How this principle feeds it

      the agent keeps executing against policies that no longer reflect the organization, and outputs stay consistent while drifting from need.

      Where is your system on this path?

      Diagnose

      Diagnose

      AI signs are on by default. Use the AI switch to exclude them. This principle applies once agents produce or act on work in your delivery.

      1. Which of these early warning signs are you seeing?

      • Agent reviews keep getting shorter, and nobody has written down why.
      • Asked if the agent is reliable, the team points to time since the last incident.
      • Nobody can say what evidence would make them look harder at the agent.
      • Sign-off on agent outputs became a formality as the agent built a track record.
      • Monitoring dashboards get checked less often than they did at deployment.
      • End users find agent failures that internal review used to catch.
      • The agent's governing policies haven't been revisited since deployment, though the organization has changed.
      • Vendor model updates go live without anyone re-checking the agent's outputs.

      2. Over the last few months, have you noticed…

      For each consequence: Not seen / Sometimes / Persistently / Not sure.

      • Reviews thinned by past success miss early signals of serious agent failures.
      • Post-incident reviews find the warning signs were on record, and nobody looked for them.
      • Recovery is slow because the team no longer knows how the agent really behaves.
      • After a big failure, trust collapses into manual processes that hurt more than calibrated oversight.
      • Agent outputs pass every check but no longer fit what the organization needs.

      3. The condition it becomes

      Based on your answers, recognize at most two conditions: No / Partly / Yes, that is us.

      Oversight Erosion: The review process still runs. It stopped meaning anything.

      Governance Drag: Work is done. Approvals are not.

      Intent Drift: The team built exactly what was asked. It is no longer what is needed.

      Take your Diagnosis to the next level by invoking AI Diagnostic Coaching tool.

      What else have you observed? Optional. Shared only if you choose to start coaching.

      Start a Diagnosis session: review and edit a draft containing your note, conditions you recognized in Step 3 (including whether they match partly), warning signs and consequence ratings before sending it. No scores or principle files are sent.

      See what practices will help you