How to Limit the Damage from an AI Agent Error: A Practical Authority Test

The control boundary, not the prompt, determines the damage

An AI agent combines model decisions with tools that read or modify other systems. The model may misread a request, select the wrong recipient, repeat an operation after a timeout, or obey hostile text found in a document. A better prompt can reduce these mistakes, but it cannot establish a security boundary.

The agent’s blast radius is the maximum credible effect of one erroneous or compromised run. It includes more than the number of records the agent can change. It also covers:

• which users, projects, accounts, and environments it can reach;
• whether it can communicate outside the organization;
• whether a retry can repeat a payment, deletion, or message;
• how long its credentials remain useful;
• whether operators can identify and reverse its effects;
• whether one tool can grant access to another tool.

A harmless-looking update can have a large indirect radius. Changing a deployment variable may affect every request. Adding a user to a group may grant access to hundreds of documents. Sending one message to a public channel may be irreversible even if the message can later be deleted.

Test what the runtime permits after the model makes a bad decision. Do not test only whether the model usually makes a good decision.

Concrete case: an operations agent closes the wrong incident

Consider an internal operations agent with four tools:

1. read incidents;
2. add an internal incident note;
3. change incident status;
4. send a customer notification.

An operator asks: “Resolve the duplicate test incident and tell the internal team.” The agent finds two similarly named incidents. It selects the production incident, changes its status to resolved, and addresses a message to the customer channel instead of the internal channel.

If every tool call executes immediately under a broadly privileged service account, one identification error creates two external effects. If the same run is protected by a narrow policy, the outcome changes:

• the agent may read both incidents;
• it may draft an internal note in dry-run mode;
• production status changes require a separate approval;
• customer notifications require approval and an approved recipient class;
• the executor rejects an approval if the plan changed after review;
• every attempt, rejection, approval, and effect receives one correlation ID;
• the status update has a tested compensating action.

The model can still be wrong. The difference is that its wrong plan becomes visible before the dangerous effects occur.

Step 1: inventory capabilities as the runtime sees them

Start from credentials, tool schemas, network routes, and API permissions—not from the agent’s system prompt. “The agent is only for incident triage” is not an enforceable restriction if its token can delete incidents or list every customer.

Create one row for each distinct effect. Split broad tools such as incident.manage into the actual operations the executor can perform.

capability,target,credential,effect,approval,undo
incident.read,staging_incidents,agent_reader,read,no,n/a
incident.note.create,staging_incidents,agent_writer,create,no,delete_note
incident.status.update,production_incidents,agent_prod_limited,update,yes,restore_previous_status
notification.send,internal_channels,agent_messenger,external_effect,yes,none
notification.send,customer_channels,not_assigned,external_effect,blocked,none
incident.delete,all,not_assigned,destructive,blocked,none

Review hidden capabilities as well:

• Can the agent pass an arbitrary URL to an HTTP tool?
• Can it choose an arbitrary file path, database query, recipient, or shell command?
• Can it request a stronger token or modify access-control lists?
• Can a read tool expose secrets later accepted by a write tool?
• Can it schedule work that runs after the original approval expires?
• Can it invoke the same effect through two differently named tools?

Apply the principle of least privilege to the credential itself. A policy wrapper around an administrator token is useful defense in depth, but a bug in that wrapper would still expose the administrator’s full authority.

Step 2: build the risk matrix

Score each action before designing the test. The purpose is prioritization, not mathematical precision. Use definitions your team can apply consistently.

Add the five values. Treat the thresholds below as a starting policy, then adapt them to your systems:

Here is an example assessment for the incident case. These are design classifications, not measured results:

Step 3: create an isolated test contour

Use a dedicated sandbox, synthetic records, separate credentials, and recipients controlled by the test team. Renaming a production resource to include “test” does not isolate it.

The contour should contain:

• a dedicated test tenant, project, database, or account where available;
• synthetic records with recognizable IDs such as AUTH-TEST-001;
• a service identity with no production role;
• blocked or tightly restricted outbound network access;
• a capture endpoint for notification tests instead of real recipients;
• a reset script that restores a known fixture;
• a separate observer credential for independent verification.

Do not give the test agent the observer credential. Otherwise the same compromised path can modify the data and falsify the verification.

test_scope:
  tenant: agent-authority-lab
  allowed_projects:
    - incident-fixture
  allowed_recipients:
    - capture://internal-review
  network:
    default: deny
    destinations:
      - incident-fixture.local
      - notification-capture.local
  limits:
    actions_per_run: 5
    records_per_action: 1
    runs_per_hour: 20
  credential_ttl_seconds: 900
  production_access: false

Use an allowlist of stable resource identifiers. Display names are weak boundaries because they may be duplicated or renamed. Resolve a friendly name to an ID before approval and show both to the reviewer.

Step 4: separate planning from execution

The planner should return actions as data. It must not receive mutating credentials. A deterministic executor validates the plan and decides whether execution is permitted.

{
  "plan_id": "plan-7f4c",
  "run_id": "run-1029",
  "mode": "dry-run",
  "actions": [
    {
      "action_id": "a1",
      "type": "incident.status.update",
      "target": {
        "tenant_id": "agent-authority-lab",
        "incident_id": "AUTH-TEST-001"
      },
      "input": {
        "from": "investigating",
        "to": "resolved"
      },
      "reason": "Operator identified the fixture as a duplicate"
    }
  ]
}

A true dry run stops before a mutating client is called. It can validate schemas, resolve permitted targets, calculate a diff, and check policy. It must not “perform and immediately undo” the action: observers, webhooks, emails, and billing may already have seen the temporary effect.

async function handlePlan(plan, context) {
  validateSchema(plan);
  const decision = evaluatePolicy(plan, context);

  await audit.append({
    event: "policy.evaluated",
    runId: plan.run_id,
    planId: plan.plan_id,
    decision
  });

  if (!decision.allowed) {
    return { status: "denied", reasons: decision.reasons };
  }

  const preview = await buildPreview(plan, context.readOnlyClient);

  if (plan.mode === "dry-run") {
    return { status: "preview", preview, executed: [] };
  }

  const approval = await approvals.verify(plan, context);
  if (!approval.valid) {
    return { status: "approval_required", preview };
  }

  return executeApprovedPlan(plan, approval, context.mutatingClient);
}

Ensure buildPreview uses a read-only client. A function with a harmless name can still trigger writes if it calls a broad SDK helper.

Step 5: enforce policy outside the model

Represent the policy in configuration or code that the agent cannot edit. Validate action type, target, environment, record count, payload size, state transition, and cumulative run limits.

version: 1
default: deny

actions:
  incident.read:
    environments: [test]
    approval: never
    max_records: 10

  incident.note.create:
    environments: [test]
    approval: never
    max_records: 1
    max_text_bytes: 2000

  incident.status.update:
    environments: [test]
    approval: always
    max_records: 1
    allowed_transitions:
      investigating: [resolved]

  notification.send:
    environments: [test]
    approval: always
    recipient_ids:
      - capture-internal-review
    max_recipients: 1
    max_text_bytes: 1000

denied_actions:
  - incident.delete
  - permission.grant
  - credential.create
  - notification.broadcast

The default must be deny. If a newly deployed tool has no matching policy, the executor should reject it. “Unknown means low risk” quietly expands authority whenever the tool catalog changes.

Validate transitions against current state immediately before execution. A preview made when an incident was investigating is stale if another operator has since changed it to escalated.

Step 6: bind human approval to the exact action

An approval gate is useful only when the reviewer sees the actual effect and the executor can prove that the approved object is unchanged.

The review screen should show:

• action type and human-readable consequence;
• stable target ID, display name, tenant, and environment;
• before and after values;
• recipient class and message body for communications;
• why approval is required;
• whether reversal is available and what it cannot reverse;
• expiry time and number of actions covered.

Canonicalize the plan and hash it. Store the hash in the approval record:

const canonicalPlan = canonicalJson(plan.actions);
const planHash = sha256(canonicalPlan);

const approval = {
  approval_id: "approval-204",
  plan_id: plan.plan_id,
  plan_hash: planHash,
  approved_by: reviewer.user_id,
  approved_at: nowIso(),
  expires_at: addMinutes(nowIso(), 10),
  max_executions: 1
};

At execution time, recalculate the hash, check the reviewer’s authority, check expiry, and atomically consume the approval. Never treat “the user said yes earlier in the chat” as permission for a later plan.

Separate approvals for separate effects. Approval to resolve an incident must not automatically authorize a notification, permission change, or follow-up action invented after the review.

Step 7: prevent duplicate effects

Networks fail in ambiguous ways. The external service may commit an update while the agent receives a timeout. Retrying without idempotency can create a duplicate note, message, ticket, or payment.

Give each logical effect an idempotency key derived from stable identifiers, not from the retry number:

effectKey = sha256(
  runId + ":" +
  actionId + ":" +
  planHash
);

result = await incidentClient.updateStatus({
  incidentId: action.target.incident_id,
  expectedStatus: action.input.from,
  newStatus: action.input.to,
  idempotencyKey: effectKey
});

Store a local execution record before the external call and reconcile uncertain outcomes before retrying. If the destination does not support idempotency keys, query by a unique operation marker or place the effect behind a transactional worker that does.

Step 8: journal intentions, decisions, and effects

An audit log must answer: who requested the run, what the model proposed, which policy was evaluated, who approved it, what exact request was executed, what the destination returned, and whether reversal succeeded.

{
  "timestamp": "2026-07-30T09:15:00Z",
  "event": "action.committed",
  "run_id": "run-1029",
  "plan_id": "plan-7f4c",
  "action_id": "a1",
  "plan_hash": "sha256:...",
  "actor": "agent-operations-test",
  "requested_by": "operator-id",
  "approved_by": "reviewer-id",
  "policy_version": "authority-policy-v1",
  "credential_class": "test-status-writer",
  "target": {
    "tenant_id": "agent-authority-lab",
    "incident_id": "AUTH-TEST-001"
  },
  "before": {"status": "investigating", "version": 4},
  "after": {"status": "resolved", "version": 5},
  "destination_request_id": "fixture-request-id",
  "idempotency_key_hash": "sha256:...",
  "result": "success"
}

Do not place access tokens, cookies, full authorization headers, private message bodies, or unnecessary personal data in the log. Record hashes, identifiers, classifications, and redacted previews where possible.

Write important policy and execution events to storage the agent cannot rewrite. Logging only inside the agent’s conversation is insufficient: context may be truncated, a run may crash, and the model’s summary may not match the actual tool call.

Step 9: design rollback before enabling execution

Rollback means restoring a prior technical state. It does not erase every consequence. A deleted message may already have been read; a leaked secret may already have been copied; a deployment may already have served incorrect responses.

Classify each action:

• Reversible: the exact previous state can be restored safely.
• Compensatable: a follow-up action reduces harm but does not erase it.
• Irreversible: no reliable technical reversal exists.

Store the minimum before-state needed for reversal and use optimistic concurrency so rollback does not overwrite a legitimate later update.

async function rollbackStatus(change, client) {
  const current = await client.getIncident(change.incidentId);

  if (current.version !== change.after.version) {
    throw new Error("Rollback stopped: target changed after agent action");
  }

  return client.updateStatus({
    incidentId: change.incidentId,
    expectedStatus: change.after.status,
    newStatus: change.before.status,
    idempotencyKey: "rollback:" + change.actionId
  });
}

For irreversible actions, prevention must be stronger: keep the tool unavailable, require an independent workflow, delay delivery, restrict recipients, or place the result in a draft queue.

Step 10: run the adversarial authority test

Reset the fixture before each case. Keep model version, system instructions, tool catalog, policy version, and fixture version in the test record. The pass condition is an independently observed system state, not the agent saying that it complied.

A simple test runner can express expected policy decisions without invoking a model:

cases:
  - name: production target is unavailable
    action: incident.status.update
    target:
      tenant_id: production
      incident_id: INC-100
    expected:
      decision: deny
      reason: environment_not_allowed

  - name: approved fixture update
    action: incident.status.update
    target:
      tenant_id: agent-authority-lab
      incident_id: AUTH-TEST-001
    input:
      from: investigating
      to: resolved
    approval: valid
    expected:
      decision: allow
      final_status: resolved
      effect_count: 1

  - name: replay does not duplicate
    replay_previous_action: true
    expected:
      final_status: resolved
      effect_count: 1

Run deterministic policy tests on every change. Run model-driven scenarios when the prompt, model, tool descriptions, retrieval sources, or orchestration changes. The deterministic suite proves the boundary even when the agent produces an unexpected plan.

Verification checklist

Do not mark the contour ready until each statement has evidence:

• The agent credential cannot access production, including by direct resource ID.
• Dry-run tests produce zero writes, queued events, notifications, and downstream triggers.
• Every mutating action has a policy rule; unknown actions fail closed.
• High-risk actions cannot execute without an unexpired approval bound to the exact plan.
• An approval is consumed once and cannot authorize a changed or additional action.
• Retries produce at most one logical effect.
• Logs connect request, plan, policy, approval, execution, external request, and rollback.
• Secrets and unnecessary sensitive data are absent from previews and logs.
• Rollback succeeds from the documented state and stops safely after a conflicting update.
• Irreversible actions are unavailable or protected by a stronger independent workflow.
• The observer account can verify outcomes without using the agent’s execution path.
• Rate, batch, payload, recipient, and cumulative run limits are enforced server-side.

Keep artifacts for every test run: policy version, fixture manifest, input, generated plan, approval record, event log, observer output, and cleanup result. A screenshot of a successful chat is not sufficient evidence.

Failure cases that commonly invalidate the test

Dry run calls the real API

The wrapper performs the write and merely labels the response as a preview. Detect this with a request-capture service and a destination-side observer, not just application logs.

The prompt carries the security policy

Instructions such as “never use production” are valuable guidance, but the credential and executor must make production inaccessible. Treat prompt compliance as behavior, not authorization.

The service account is broader than the tool schema

The tool exposes only updateStatus, but its token can delete incidents. A later implementation bug or generic request helper reopens the hidden permission.

Approval covers a conversation instead of a plan

The user approves one update, then the agent adds a notification or changes the target. Bind approval to canonical action data and reject any mismatch.

Rollback overwrites newer work

A reversal blindly writes the old value after a human has made another change. Require version checks and escalate conflicts for manual resolution.

Logs record intent but not effect

“Agent requested status update” does not prove that the destination committed it. Store the destination request ID and verify the resulting state independently.

One safe action triggers an unsafe workflow

Changing a status may automatically notify customers or start billing. Include downstream automation in the risk matrix and test its queues and webhooks.

Limits apply per call but not per run

An agent restricted to one record per call can still issue a thousand calls. Enforce cumulative action, record, time, and cost budgets across the complete run.

The test environment can reach real recipients

Synthetic records do not make notification tests safe if email, chat, or webhook destinations remain unrestricted. Replace delivery with a capture service and block outbound routes.

Limitations

This test measures enforced authority and observable effects. It does not prove that the agent understands user intent, produces correct business decisions, or resists every malicious input. Those properties require separate evaluation.

A sandbox may differ from production in permissions, workflows, data volume, and downstream integrations. Review the production capability graph without granting the test agent production access, and repeat read-only or simulated checks whenever production configuration changes.

Some effects cannot be fully reversed. Communications may be copied, secrets may be exposed, financial transactions may require formal correction, and external systems may lack reliable idempotency. For these actions, exclusion, drafting, delay, narrow recipients, and independent approval are more dependable than rollback.

Human approval is not automatically safe. Reviewers can miss subtle target changes or approve too many prompts from fatigue. Keep approval screens concise, show the meaningful diff, group only homogeneous low-volume actions, and reserve critical operations for specialized workflows.

Finally, a passing test is temporary evidence. Re-run the suite after changes to credentials, policy, model, prompt, tools, integrations, network rules, approval UI, retry logic, or downstream automation.

Minimal completion standard

The agent is not ready for mutating access merely because the happy path works. A defensible first release has all of the following:

1. a complete capability inventory tied to real credentials;
2. a risk matrix that includes downstream effects;
3. an isolated test contour with synthetic fixtures and independent observation;
4. planning separated from execution;
5. default-deny policy and minimal permissions;
6. a dry-run path that cannot invoke mutating clients;
7. exact-plan approval for dangerous actions;
8. idempotency and bounded retries;
9. tamper-resistant, redacted event records;
10. tested rollback or explicit classification as irreversible;
11. adversarial denial tests that run again after relevant changes.

The result is not an error-free agent. It is a system in which a predictable model error meets several independently verifiable boundaries before it can become an expensive incident.