Agent workflows · Intermediate

How to Build a Harness for a Repeating Agent Workflow

Как построить harness для повторяющегося workflow агента
Temporary fallback cover; replace in editorial pass.

A one-off agent task becomes dependable only when its boundaries are explicit: the input is known, each step leaves evidence, completion is testable, and the run can stop before time or budget disappears.

15 min read Updated 21 November 2026 By Agent Lab Journal

An agent can often produce a useful answer from a short instruction. That is not the same as having a repeatable workflow. Repetition introduces questions that a one-off prompt can hide: Which sources were consulted? What counts as complete? What happens when a page is unavailable? How many retries are allowed? How do you compare two runs when their outputs differ?

This article builds a small harness: a wrapper that supplies controlled inputs, records intermediate evidence, applies limits, and evaluates the result. The example is intentionally ordinary: collect a short list of public documentation pages, extract specified fields, and produce a structured report. The method also applies to recurring research, support triage, data extraction, and content audits.

The concrete case

Suppose a team needs a weekly report about five product pages. For each page, the workflow should record the page title, the last visible update date if present, three relevant claims, and the URL. It must not invent missing values. It should return a report even if one page fails, and it should stop after a fixed number of pages, requests, and model calls.

The first version might be a single instruction: “Review these pages and summarize changes.” That version is difficult to test because it mixes discovery, browsing, interpretation, formatting, and error handling. The harness separates those concerns.

Workflow contract for the example
Part Contract
Input A versioned list of HTTPS URLs and a reporting date.
Output One record per input URL, plus run metadata and failures.
Completion Every input has a result, an explicit failure, or a skipped status.
Limits Five pages, ten retrieval attempts, and six model calls.
Evidence A retrieval ledger with URL, status, timestamp, and content hash.

1. Make the input explicit

Start with a small input object rather than an informal prompt. Keep secrets, personal data, and credentials out of fixtures. Use placeholder domains or public pages that you are permitted to retrieve.

{
  "run_id": "docs-audit-2026-11-21-001",
  "report_date": "2026-11-21",
  "targets": [
    {"id": "page-001", "url": "https://example.com/docs/one"},
    {"id": "page-002", "url": "https://example.com/docs/two"}
  ],
  "fields": ["title", "updated_date", "claims"],
  "limits": {
    "max_targets": 5,
    "max_retrieval_attempts": 10,
    "max_model_calls": 6,
    "max_output_tokens": 2500
  }
}

The run_id makes a result addressable. Stable target IDs let you compare runs even when the order of URLs changes. The field list prevents the agent from silently expanding the task. A fixture is a saved input used for repeatable testing; keep at least one small, deterministic fixture alongside any live input.

2. Split the work into bounded steps

Each step should have one purpose and a checkable result. A useful minimum sequence is:

  1. Validate the input schema and enforce the target limit.
  2. Retrieve each target and record the response metadata.
  3. Normalize the retrieved content into the format the model needs.
  4. Extract only the requested fields.
  5. Validate the structured output.
  6. Write the report and the run ledger.

Do not ask the model to decide whether the run is complete. The harness should know that all target IDs have a terminal status. The model can help interpret content, but deterministic code should own counting, schemas, limits, and persistence.

validate_input
  - reject missing URL or target id
  - reject more than max_targets

for target in targets:
  retrieve(target)
  record_ledger_entry(target)

  if retrieval_succeeded:
    extract_requested_fields(target)
    validate_record(target)
  else:
    record_failure(target)

assert every target has a terminal status
write report
write ledger

3. Define completion before writing prompts

A completion criterion is a rule that can be evaluated from the run state. “The summary looks good” is useful human feedback, but it is not sufficient as the only stopping condition.

For the example, a run is complete when:

  • the input passed validation;
  • each target has success, failed, or skipped;
  • successful records contain exactly the requested top-level fields;
  • missing source values are represented as null or an explicit “not found” value;
  • the report and ledger are written together; and
  • no configured limit was exceeded.

This distinction matters when a target fails. A partial run can still be complete if the failure is recorded and the stopping rule has been met. “Complete” describes the process state; it does not mean every target succeeded.

4. Add a retrieval ledger

A retrieval ledger turns “the agent saw some pages” into inspectable evidence. Record enough to reproduce the decision without storing more content than you need. Avoid putting cookies, authorization headers, personal data, or full sensitive page bodies into logs.

{
  "run_id": "docs-audit-2026-11-21-001",
  "target_id": "page-001",
  "url": "https://example.com/docs/one",
  "attempt": 1,
  "started_at": "2026-11-21T09:00:00Z",
  "finished_at": "2026-11-21T09:00:01Z",
  "status": "ok",
  "http_status": 200,
  "content_type": "text/html",
  "content_sha256": "example-hash",
  "bytes_read": 18420,
  "error": null
}

The hash in this example is a placeholder, not a measured value. In a real run, calculate it from the exact normalized content used by the extraction step. That makes it possible to tell whether two reports used the same material without treating a URL as proof that its content was unchanged.

Keep the ledger append-only during a run. If a retry occurs, add another attempt instead of overwriting the first result. This makes timeout patterns, rate limits, and transient failures visible.

5. Control retries and costs

A retry is a new attempt, not free persistence. Define a maximum per target and a maximum for the complete run. Use idempotency where a repeated operation could create a duplicate report or external side effect. For a read-only audit, the simplest safe policy is usually to retry retrieval but not to repeat the final write blindly.

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}
MAX_RETRIEVAL_ATTEMPTS = 2
MAX_MODEL_CALLS = 6

def can_retry(response, attempt, run_state):
    return (
        response.status in RETRYABLE_STATUS
        and attempt < MAX_RETRIEVAL_ATTEMPTS
        and run_state.retrieval_attempts < 10
    )

Cost control is easier when the harness counts observable units: retrieval attempts, model calls, input bytes, output tokens, elapsed time, and completed targets. If your provider exposes usage data, store it with the run. If it does not, record the counters you can measure and label any cost estimate as an estimate.

A useful budget policy is fail-closed: once a hard limit is reached, stop starting new work, mark remaining targets as skipped, and write the partial report. This is safer than allowing an agent to “try one more thing” without a bound.

6. Constrain the model output

Give the model the smallest context that can answer the requested fields. Ask for a schema, not prose that another component must guess how to parse. The extraction prompt can be simple:

Extract the requested fields from the supplied page text.

Rules:
- Use only the supplied text.
- Do not infer a date that is not present.
- Return null when a field is absent.
- Include at most three claims.
- Each claim must include a short supporting quote or null.
- Return valid JSON matching this schema:

{
  "title": "string|null",
  "updated_date": "string|null",
  "claims": [
    {"text": "string", "evidence": "string|null"}
  ]
}

The harness must still validate the response. A valid JSON document can contain the wrong fields, too many claims, or invented evidence. Reject, repair, or quarantine invalid records according to a policy you can observe. Do not silently convert an invalid response into a successful one.

7. Compare runs by cost and quality

A repeating workflow needs a comparison record. Store the configuration version, fixture ID, model identifier if applicable, counters, statuses, and evaluation results. Do not compare raw output alone: a shorter answer may be cheaper but omit required fields.

Minimal run summary
Metric Why it matters
Targets attempted Shows the actual workload.
Successful records Measures usable coverage.
Retries and failures Shows operational stability.
Model calls and tokens Supports cost comparison when usage is available.
Schema-valid records Separates parseability from apparent fluency.
Human review flags Captures issues automated checks cannot establish.

For a controlled experiment, hold the fixture and task contract constant, change one configuration at a time, and record the result. This does not prove that one configuration is universally better. It gives you a bounded comparison under the conditions you recorded.

Verification checklist

Before relying on a harness, test the workflow with deterministic fixtures and failure injection. The following checks are practical because each has an observable expected result:

  • An empty target list is rejected or produces a documented empty run.
  • A sixth target is not retrieved when the limit is five.
  • A timeout creates a ledger entry and a terminal failure status.
  • A retry adds a new attempt instead of overwriting the old one.
  • Malformed model JSON cannot become a successful record silently.
  • A missing source date remains null or “not found,” not a guessed date.
  • Re-running the same fixture produces comparable counters and the same validation outcome.
  • When the budget is exhausted, remaining work is skipped and the report is written.

These are test cases to run, not results claimed by this article. The article does not execute your retriever, model provider, or storage layer.

Common failure cases

The agent keeps exploring

Add an explicit target set and a maximum number of retrieval attempts. Discovery can be a separate workflow with its own contract. Do not let an extraction run become open-ended research by accident.

The report is complete but not trustworthy

Require evidence fields and preserve the normalized-content hash. Then inspect a sample of records against the ledger. A schema check proves shape, not truth.

Retries multiply the bill

Count retries globally, add a deadline, and stop on the hard budget. Record why a retry happened. If a failure is deterministic, retrying it is usually wasted work.

A provider change breaks comparisons

Include the workflow version, prompt version, model identifier, and fixture identifier in the run summary. If any of these change, treat the comparison as a new experiment.

Logs expose more than intended

Redact secrets before logging, minimize stored page text, restrict access to ledgers, and define a retention period. A reproducible process does not require permanent storage of every raw input.

Limitations

A harness cannot make an unreliable source reliable, remove ambiguity from a vague task, or guarantee factual correctness from an extraction model. Retrieval status and content hashes also do not prove that a page was authoritative. Human review may still be necessary for high-impact decisions.

Cost comparisons are conditional on the measured configuration and usage accounting available in your environment. If token usage, provider pricing, or infrastructure time is not recorded, report counters instead of fabricating a currency total.

Final pattern

The smallest useful harness has five properties: explicit inputs, bounded steps, machine-checkable completion, hard limits, and an evidence ledger. Add structured output validation and run summaries, then test the failure paths deliberately. Once those pieces exist, a one-off agent task becomes a process you can repeat, inspect, and compare without pretending that uncertainty has disappeared.

Continue with the practical guides or look up related terms in the glossary.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →