ANALYSIS OF A REAL-WORLD ERROR

How we identified unnecessary AI agent costs

The agent completed its tasks, but the budget was being spent faster than expected, and the final report did not explain why. The cause was neither the model’s price nor a single long request: the same stage was being triggered again unnoticed after a timeout. We reconstructed the sequence from the event log and added an operation identifier, duplicate protection, and a separate cost report for each task.

What exactly was happening

The workflow looked simple: receive an event, collect data, call the model, save the response, and mark the task as completed. On the surface, everything worked—the user received one result. Internally, however, some tasks generated several calls to the model.

The repeated request occurred at the boundary between making the request and saving the state. The model managed to process the request, but our worker either did not receive the response in time or did not manage to record the status. The queue treated the attempt as failed and started it again. The provider had already accepted the first request, so every new attempt became a separate billable operation.

event received
  → task taken from the queue
  → request sent to the model
  → wait exceeded the limit
  → task returned to the queue
  → the same request sent again
  → only the last response saved

The final report stored the task result but not the history of attempts. As a result, the cost appeared as an unexplained discrepancy: one visible response, but several actual requests.

Why a conventional log was not enough

The log contained messages saying “task started” and “task completed,” but there was no shared identifier connecting the incoming event, the attempt, the model call, and the charge. Such a log shows activity, but it does not provide observability.

For the investigation, we needed four different identifiers:

• event_id — a unique incoming event;
• job_id — the agent’s logical task;
• attempt — the execution attempt number;
• request_id — an individual call to the model.

If you store only job_id, repeated attempts are merged together. If you store only request_id, it is unclear which user task the request served. Both levels are required for cost control.

Step 1. Count requests before aggregation

Do not record only the daily total. First, create one row for every call to the model. The minimum set of fields is:

{
  "event_id": "evt_example",
  "job_id": "job_example",
  "attempt": 1,
  "request_id": "req_example",
  "stage": "summarize",
  "model": "configured-model",
  "input_tokens": 0,
  "output_tokens": 0,
  "cost": null,
  "status": "started",
  "started_at": "ISO-8601 timestamp",
  "finished_at": null,
  "error_type": null
}

Populate token counts and cost from the provider’s response or from a separate billing event. Do not calculate a financial report solely from text length: it may be a useful estimate, but it is not a confirmed charge.

The started status must be saved before the network call. After receiving the response, update the same record instead of creating a new one. This keeps an interrupted attempt visible so that it can be matched to the next run.

Step 2. Find repeated operations

Group the log by task and stage. For example, for SQLite or PostgreSQL, the basis of the check might look like this:

SELECT
  job_id,
  stage,
  COUNT(*) AS request_count,
  SUM(COALESCE(cost, 0)) AS total_cost
FROM model_requests
GROUP BY job_id, stage
HAVING COUNT(*) > 1
ORDER BY request_count DESC;

The presence of multiple requests does not by itself prove that there is an error. The agent may deliberately call the model for planning, verification, and the final response. What is suspicious is a repeated request with the same job_id, stage, and normalized input.

To compare inputs without storing sensitive text, calculate a fingerprint after stable normalization:

input_hash = sha256(
  model + "\n" +
  stage + "\n" +
  normalized_system_prompt + "\n" +
  normalized_user_input
)

Do not include the current time, random identifiers, or a changing order of service fields in the normalization. Otherwise, two requests that are identical in meaning will receive different fingerprints.

Step 3. Distinguish repeated events from repeated attempts

We had two potential sources of duplicates. The first was repeated delivery of an incoming webhook. The second was restarting an already-created task after a timeout. Both boundaries need protection.

At the input boundary, create a unique constraint on the source and event identifier:

CREATE UNIQUE INDEX uniq_source_event
ON inbound_events(source, event_id);

Before a billable stage, build an operation key from the task, stage, and input fingerprint. One key must correspond to one logical result:

operation_key = sha256(job_id + ":" + stage + ":" + input_hash)

Store the key in a table with a unique index. If a record already exists with the completed status, return the saved result. If the operation is still running, do not send a second request: reschedule the task for a brief status check.

Redelivering a task is safe only when the repeated delivery does not create a second billable action.

Step 4. Configure retries by error type

Unconditionally retrying every error quickly turns into a cost generator. The configuration should limit the number of attempts, the delay, and the task’s total budget:

retry:
  max_attempts: 3
  backoff_seconds: [2, 10, 30]
  retry_on:
    - rate_limit
    - temporary_unavailable
    - connection_reset
  stop_on:
    - invalid_request
    - authentication_error
    - context_limit

budget:
  max_model_requests_per_job: 4
  max_input_tokens_per_job: 30000
  require_approval_above_configured_cost: true

The values shown here are an example of the structure, not universal limits. Use constraints based on your own load model and pricing plan. A format error or exceeding the allowed context limit generally needs to be corrected rather than resent unchanged.

If the task reaches its limit, stop it in the budget_exceeded state. For an expensive continuation, use an approval gate: the agent shows the attempts already made, the accumulated cost, and the reason for the next request.

Step 5. Create a clear report

A final total without a breakdown is of little help during debugging. The task report should answer four questions:

• how many logical stages were planned;
• how many requests were actually sent;
• which requests were retries and why;
• which costs are confirmed and which are still only estimates.

Task: job_example
Status: completed
Stages: plan, retrieve, summarize
Model requests: 4
Repeated attempts: 1
Reason for retry: temporary_unavailable
Confirmed cost: value from billing
Unfinished requests: 0

Do not treat zero cost and unknown cost as the same thing. For a request without billing data, store null or the pending status. Otherwise, the report will appear precise even though some costs have not yet been accounted for.

How to test duplicate protection

It is best to run the test in a test project with minimal limits. Do not call an expensive model merely to simulate a failure: you can replace the network client with a stub that records the number of calls.

1. Submit the same incoming event twice. Only one logical task should remain in the database.
2. Interrupt execution after receiving the response but before the worker completes. Restarting the task must not create a second completed operation.
3. Return a temporary error before the request is accepted. The next attempt should receive a new number while retaining the same job_id.
4. Return a permanent format error. The agent should stop without retrying.
5. Run the task until it reaches the request limit. The next billable stage should be blocked.
6. Compare the number of model_requests records with the provider’s log or billing data for the same interval.

The success criterion is not simply a single response in the interface. Every actual request has a record, every retry is explained, and the same event does not create a second billable action.

What can go wrong

The request completed at the provider, but the connection was interrupted. It is impossible to determine the result reliably from the local side. Use the provider’s idempotency key if supported, or check the request log before retrying.

Two workers saw the same available task simultaneously. A “read first, then write” check is not sufficient. You need a unique index, a transaction, or an atomic lock.

The input fingerprint changes with every attempt. Remove dynamic fields from normalization, or create the request content once and store it with the task.

Retries were disabled completely. This reduces costs but makes the system fragile in the event of temporary network errors. The goal is not zero retries, but controlled and explainable retries.

The limit is checked after the call. The budget must be reserved before the billable operation and adjusted after the actual usage is received.

Limitations of the approach

A local log does not replace provider data: a charge may appear later, and calculation rules depend on the pricing plan and model. A fingerprint identifies exact repeats, but it does not always recognize two slightly different requests that perform the same work. Finally, strict limits may stop a useful long-running task, so they need to be configured separately for different types of operations.

The main outcome of this analysis was that cost became part of the task state rather than a separate figure at the end of the month. Every request is now tied to the reason it was initiated, retries are constrained by rules, and the report makes it possible to understand which action consumed the budget.

← All guides · Lab glossary →