Agent workflows · Intermediate
How to Build a Harness for a Repeating Agent Workflow

A one-off agent task becomes dependable only when its boundaries are explicit: the input is known, each step leaves evidence, completion is testable, and the run can stop before time or budget disappears.
An agent can often produce a useful answer from a short instruction. That is not the same as having a repeatable workflow. Repetition introduces questions that a one-off prompt can hide: Which sources were consulted? What counts as complete? What happens when a page is unavailable? How many retries are allowed? How do you compare two runs when their outputs differ?
This article builds a small harness: a wrapper that supplies controlled inputs, records intermediate evidence, applies limits, and evaluates the result. The example is intentionally ordinary: collect a short list of public documentation pages, extract specified fields, and produce a structured report. The method also applies to recurring research, support triage, data extraction, and content audits.
The concrete case
Suppose a team needs a weekly report about five product pages. For each page, the workflow should record the page title, the last visible update date if present, three relevant claims, and the URL. It must not invent missing values. It should return a report even if one page fails, and it should stop after a fixed number of pages, requests, and model calls.
The first version might be a single instruction: “Review these pages and summarize changes.” That version is difficult to test because it mixes discovery, browsing, interpretation, formatting, and error handling. The harness separates those concerns.
| Part | Contract |
|---|---|
| Input | A versioned list of HTTPS URLs and a reporting date. |
| Output | One record per input URL, plus run metadata and failures. |
| Completion | Every input has a result, an explicit failure, or a skipped status. |
| Limits | Five pages, ten retrieval attempts, and six model calls. |
| Evidence | A retrieval ledger with URL, status, timestamp, and content hash. |
1. Make the input explicit
Start with a small input object rather than an informal prompt. Keep secrets, personal data, and credentials out of fixtures. Use placeholder domains or public pages that you are permitted to retrieve.
{
"run_id": "docs-audit-2026-11-21-001",
"report_date": "2026-11-21",
"targets": [
{"id": "page-001", "url": "https://example.com/docs/one"},
{"id": "page-002", "url": "https://example.com/docs/two"}
],
"fields": ["title", "updated_date", "claims"],
"limits": {
"max_targets": 5,
"max_retrieval_attempts": 10,
"max_model_calls": 6,
"max_output_tokens": 2500
}
}
The run_id makes a result addressable. Stable target IDs let
you compare runs even when the order of URLs changes. The field list prevents
the agent from silently expanding the task. A fixture
is a saved input used for repeatable testing; keep at least one small,
deterministic fixture alongside any live input.
2. Split the work into bounded steps
Each step should have one purpose and a checkable result. A useful minimum sequence is:
- Validate the input schema and enforce the target limit.
- Retrieve each target and record the response metadata.
- Normalize the retrieved content into the format the model needs.
- Extract only the requested fields.
- Validate the structured output.
- Write the report and the run ledger.
Do not ask the model to decide whether the run is complete. The harness should know that all target IDs have a terminal status. The model can help interpret content, but deterministic code should own counting, schemas, limits, and persistence.
validate_input
- reject missing URL or target id
- reject more than max_targets
for target in targets:
retrieve(target)
record_ledger_entry(target)
if retrieval_succeeded:
extract_requested_fields(target)
validate_record(target)
else:
record_failure(target)
assert every target has a terminal status
write report
write ledger
3. Define completion before writing prompts
A completion criterion is a rule that can be evaluated from the run state. “The summary looks good” is useful human feedback, but it is not sufficient as the only stopping condition.
For the example, a run is complete when:
- the input passed validation;
- each target has
success,failed, orskipped; - successful records contain exactly the requested top-level fields;
- missing source values are represented as
nullor an explicit “not found” value; - the report and ledger are written together; and
- no configured limit was exceeded.
This distinction matters when a target fails. A partial run can still be complete if the failure is recorded and the stopping rule has been met. “Complete” describes the process state; it does not mean every target succeeded.
4. Add a retrieval ledger
A retrieval ledger turns “the agent saw some pages” into inspectable evidence. Record enough to reproduce the decision without storing more content than you need. Avoid putting cookies, authorization headers, personal data, or full sensitive page bodies into logs.
{
"run_id": "docs-audit-2026-11-21-001",
"target_id": "page-001",
"url": "https://example.com/docs/one",
"attempt": 1,
"started_at": "2026-11-21T09:00:00Z",
"finished_at": "2026-11-21T09:00:01Z",
"status": "ok",
"http_status": 200,
"content_type": "text/html",
"content_sha256": "example-hash",
"bytes_read": 18420,
"error": null
}
The hash in this example is a placeholder, not a measured value. In a real run, calculate it from the exact normalized content used by the extraction step. That makes it possible to tell whether two reports used the same material without treating a URL as proof that its content was unchanged.
Keep the ledger append-only during a run. If a retry occurs, add another attempt instead of overwriting the first result. This makes timeout patterns, rate limits, and transient failures visible.
5. Control retries and costs
A retry is a new attempt, not free persistence. Define a maximum per target and a maximum for the complete run. Use idempotency where a repeated operation could create a duplicate report or external side effect. For a read-only audit, the simplest safe policy is usually to retry retrieval but not to repeat the final write blindly.
RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}
MAX_RETRIEVAL_ATTEMPTS = 2
MAX_MODEL_CALLS = 6
def can_retry(response, attempt, run_state):
return (
response.status in RETRYABLE_STATUS
and attempt < MAX_RETRIEVAL_ATTEMPTS
and run_state.retrieval_attempts < 10
)
Cost control is easier when the harness counts observable units: retrieval attempts, model calls, input bytes, output tokens, elapsed time, and completed targets. If your provider exposes usage data, store it with the run. If it does not, record the counters you can measure and label any cost estimate as an estimate.
A useful budget policy is fail-closed: once a hard limit is reached, stop starting new work, mark remaining targets as skipped, and write the partial report. This is safer than allowing an agent to “try one more thing” without a bound.
6. Constrain the model output
Give the model the smallest context that can answer the requested fields. Ask for a schema, not prose that another component must guess how to parse. The extraction prompt can be simple:
Extract the requested fields from the supplied page text.
Rules:
- Use only the supplied text.
- Do not infer a date that is not present.
- Return null when a field is absent.
- Include at most three claims.
- Each claim must include a short supporting quote or null.
- Return valid JSON matching this schema:
{
"title": "string|null",
"updated_date": "string|null",
"claims": [
{"text": "string", "evidence": "string|null"}
]
}
The harness must still validate the response. A valid JSON document can contain the wrong fields, too many claims, or invented evidence. Reject, repair, or quarantine invalid records according to a policy you can observe. Do not silently convert an invalid response into a successful one.
7. Compare runs by cost and quality
A repeating workflow needs a comparison record. Store the configuration version, fixture ID, model identifier if applicable, counters, statuses, and evaluation results. Do not compare raw output alone: a shorter answer may be cheaper but omit required fields.
| Metric | Why it matters |
|---|---|
| Targets attempted | Shows the actual workload. |
| Successful records | Measures usable coverage. |
| Retries and failures | Shows operational stability. |
| Model calls and tokens | Supports cost comparison when usage is available. |
| Schema-valid records | Separates parseability from apparent fluency. |
| Human review flags | Captures issues automated checks cannot establish. |
For a controlled experiment, hold the fixture and task contract constant, change one configuration at a time, and record the result. This does not prove that one configuration is universally better. It gives you a bounded comparison under the conditions you recorded.
Verification checklist
Before relying on a harness, test the workflow with deterministic fixtures and failure injection. The following checks are practical because each has an observable expected result:
- An empty target list is rejected or produces a documented empty run.
- A sixth target is not retrieved when the limit is five.
- A timeout creates a ledger entry and a terminal failure status.
- A retry adds a new attempt instead of overwriting the old one.
- Malformed model JSON cannot become a successful record silently.
- A missing source date remains null or “not found,” not a guessed date.
- Re-running the same fixture produces comparable counters and the same validation outcome.
- When the budget is exhausted, remaining work is skipped and the report is written.
These are test cases to run, not results claimed by this article. The article does not execute your retriever, model provider, or storage layer.
Common failure cases
The agent keeps exploring
Add an explicit target set and a maximum number of retrieval attempts. Discovery can be a separate workflow with its own contract. Do not let an extraction run become open-ended research by accident.
The report is complete but not trustworthy
Require evidence fields and preserve the normalized-content hash. Then inspect a sample of records against the ledger. A schema check proves shape, not truth.
Retries multiply the bill
Count retries globally, add a deadline, and stop on the hard budget. Record why a retry happened. If a failure is deterministic, retrying it is usually wasted work.
A provider change breaks comparisons
Include the workflow version, prompt version, model identifier, and fixture identifier in the run summary. If any of these change, treat the comparison as a new experiment.
Logs expose more than intended
Redact secrets before logging, minimize stored page text, restrict access to ledgers, and define a retention period. A reproducible process does not require permanent storage of every raw input.
Limitations
A harness cannot make an unreliable source reliable, remove ambiguity from a vague task, or guarantee factual correctness from an extraction model. Retrieval status and content hashes also do not prove that a page was authoritative. Human review may still be necessary for high-impact decisions.
Cost comparisons are conditional on the measured configuration and usage accounting available in your environment. If token usage, provider pricing, or infrastructure time is not recorded, report counters instead of fabricating a currency total.
Final pattern
The smallest useful harness has five properties: explicit inputs, bounded steps, machine-checkable completion, hard limits, and an evidence ledger. Add structured output validation and run summaries, then test the failure paths deliberately. Once those pieces exist, a one-off agent task becomes a process you can repeat, inspect, and compare without pretending that uncertainty has disappeared.
Continue with the practical guides or look up related terms in the glossary.