Engineering guide · Advanced · 35 min read
Building a Reproducible Eval and Release Gate for an AI Agent

An agent can say “the payment was created” while the payment provider says nothing was created. The dangerous part is not merely a bad sentence. It is a mismatch between the agent’s report and the external system’s durable state. This guide builds an offline TypeScript harness that makes that mismatch observable, repeatable, and release-blocking.
The failure we are trying to catch
Consider a support agent that receives a request to create a payment. It calls an internal payment tool, receives an exception or an ambiguous timeout, and then produces a reassuring completion message: “Payment created successfully.” A downstream operator may trust that message, ship an order, or tell a customer to wait for settlement.
The core test question is simple: Does the agent’s claimed outcome match an independently observed outcome?
This article uses evals as executable checks over agent behavior, not as a collection of subjective transcripts. Each scenario has a fixture, a controlled external system, an independent observer, and explicit pass criteria. A fixture describes the task and the expected safety properties. The observer reads provider state through a path that does not trust the agent’s own summary.
The final result is a release gate: a command that exits non-zero when a critical invariant fails. The harness does not prove that a production integration is safe. It does prove that the implementation can detect a defined class of contradiction under deterministic, offline conditions.
What “success” must mean
An agent’s natural-language response is evidence about what the model believes happened. It is not authoritative evidence about what happened in an external system. For a state-changing operation, define success as a conjunction:
success =
agent_claim_is_allowed
&& provider_state_matches_expected_state
&& observer_can_verify_that_state
&& no_critical_invariant_was_violated
The distinction matters because each term can fail independently:
- The provider may reject the request.
- The provider may accept it but return an ambiguous response.
- The agent may receive a timeout after the provider has committed the change.
- The agent may receive a failure and incorrectly report success.
- The provider may contain a record, but with the wrong amount or currency.
- The observer may be accidentally coupled to the same mock or cache used by the agent.
A useful harness therefore checks both behavior and state. It should not ask only, “Did the agent produce the expected text?” It should also ask, “What does the external system contain after the action, and can an independent reader verify it?”
The smallest useful architecture
The harness below has five boundaries:
- Task fixture: input, expected claims, and safety invariants.
- Agent under test: the code that decides what tool to call and what to report.
- Payment provider: a deterministic fake with injectable outcomes.
- Independent observer: a separate reader of provider state.
- Evaluator and gate: assertions that classify the run and decide whether release is allowed.
The provider and observer are deliberately separate interfaces. The agent receives a narrow command interface. The observer receives a read interface. In a production system these may eventually be backed by the same provider, but the test should preserve the conceptual independence: the agent reports through one path, while the evaluator verifies through another.
fixture
|
v
agent ---- command API ----> provider state
| ^
| |
+---- claimed result observer read API
|
v
evaluator ----> release gate
Project setup
The example uses a minimal TypeScript project and Node’s built-in test runner. It has no network calls, credentials, real customer data, or dependency on a payment-provider SDK.
mkdir agent-eval-gate
cd agent-eval-gate
npm init -y
npm install --save-dev typescript tsx @types/node
Add the following scripts to package.json:
{
"scripts": {
"test": "node --import tsx --test",
"gate": "node --import tsx src/gate.ts"
}
}
Create a tsconfig.json with strict checking:
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noEmit": true,
"skipLibCheck": true
},
"include": ["src"]
}
A possible file layout is:
src/
domain.ts
provider.ts
agent.ts
observer.ts
fixtures.ts
evaluator.ts
gate.ts
harness.test.ts
The code in the next sections is a complete starter harness. It is intentionally small enough to inspect in one sitting. Replace the deliberately faulty agent with the integration under evaluation, then preserve the observer and critical assertions.
1. Define domain types and invariants
Start with explicit types. Avoid using a free-form string such as "success" for every state.
A small state model makes contradictory outcomes visible.
// src/domain.ts
export type Currency = "USD" | "EUR";
export type PaymentStatus =
| "created"
| "pending"
| "succeeded"
| "failed"
| "cancelled";
export interface Payment {
id: string;
amountMinor: number;
currency: Currency;
status: PaymentStatus;
}
export type ProviderFailure =
| "none"
| "timeout-before-commit"
| "timeout-after-commit"
| "rejected"
| "malformed-response";
export interface CreatePaymentRequest {
amountMinor: number;
currency: Currency;
idempotencyKey: string;
}
export interface ProviderResponse {
kind: "accepted" | "rejected" | "ambiguous";
payment?: Payment;
errorCode?: string;
}
export interface AgentReport {
text: string;
claimedStatus: "success" | "pending" | "failed" | "unknown";
paymentId?: string;
}
export interface TaskFixture {
name: string;
request: CreatePaymentRequest;
expected: {
providerStatus: PaymentStatus | "absent";
allowedClaimedStatuses: Array<AgentReport["claimedStatus"]>;
};
critical: boolean;
failureMode: ProviderFailure;
}
The important design choice is that the expected provider status is separate from the allowed agent claim. For example, after a timeout before commit, the provider record may be absent and the agent must not claim success. After a timeout after commit, the provider record may exist, but the agent should report uncertainty or pending state until it performs a safe lookup.
2. Build a deterministic payment provider
The provider fake stores state in memory and exposes controlled failure modes. A deterministic fake is not a simulation of every provider behavior. It is a compact test double for the failure contracts that matter to this article.
// src/provider.ts
import {
CreatePaymentRequest,
Payment,
ProviderFailure,
ProviderResponse
} from "./domain.js";
export interface PaymentCommandApi {
createPayment(request: CreatePaymentRequest): Promise<ProviderResponse>;
}
export interface PaymentReadApi {
getPayment(id: string): Promise<Payment | undefined>;
}
export class FakePaymentProvider
implements PaymentCommandApi, PaymentReadApi {
private readonly payments = new Map<string, Payment>();
private readonly keys = new Map<string, string>();
private sequence = 0;
constructor(private readonly failure: ProviderFailure = "none") {}
async createPayment(
request: CreatePaymentRequest
): Promise<ProviderResponse> {
const existingId = this.keys.get(request.idempotencyKey);
if (existingId) {
const existing = this.payments.get(existingId);
return existing
? { kind: "accepted", payment: structuredClone(existing) }
: { kind: "ambiguous" };
}
if (this.failure === "timeout-before-commit") {
throw new Error("provider_timeout_before_commit");
}
if (this.failure === "rejected") {
return { kind: "rejected", errorCode: "card_declined" };
}
const payment: Payment = {
id: `pay_test_${++this.sequence}`,
amountMinor: request.amountMinor,
currency: request.currency,
status: "created"
};
this.keys.set(request.idempotencyKey, payment.id);
this.payments.set(payment.id, payment);
if (this.failure === "timeout-after-commit") {
throw new Error("provider_timeout_after_commit");
}
if (this.failure === "malformed-response") {
return { kind: "accepted" };
}
return {
kind: "accepted",
payment: structuredClone(payment)
};
}
async getPayment(id: string): Promise<Payment | undefined> {
const payment = this.payments.get(id);
return payment ? structuredClone(payment) : undefined;
}
snapshot(): Payment[] {
return [...this.payments.values()].map((payment) => structuredClone(payment));
}
}
This fake includes one particularly important case: timeout-after-commit. The provider writes
the payment and then throws. That models a network failure after the external system has accepted the
command but before the client receives a usable response. A retry without an idempotency key could create
a duplicate. A report of definite success without a lookup could still be premature.
3. Add the agent under test
The agent receives a command API and returns a structured report. The report can include text for a human, but the evaluator should rely on structured fields wherever possible.
// src/agent.ts
import {
AgentReport,
CreatePaymentRequest
} from "./domain.js";
import { PaymentCommandApi } from "./provider.js";
export interface Agent {
createPayment(request: CreatePaymentRequest): Promise<AgentReport>;
}
export class PaymentAgent implements Agent {
constructor(private readonly provider: PaymentCommandApi) {}
async createPayment(
request: CreatePaymentRequest
): Promise<AgentReport> {
try {
const response = await this.provider.createPayment(request);
if (response.kind === "accepted" && response.payment) {
return {
text: `Payment ${response.payment.id} created.`,
claimedStatus: "success",
paymentId: response.payment.id
};
}
if (response.kind === "rejected") {
return {
text: "The payment was rejected.",
claimedStatus: "failed"
};
}
return {
text: "The payment result is uncertain.",
claimedStatus: "unknown"
};
} catch {
return {
text: "The payment was created successfully.",
claimedStatus: "success"
};
}
}
}
The catch block is intentionally unsafe. It converts an exception into a definite success claim. The test suite should expose that behavior before this implementation reaches a release branch.
Notice that the agent does not receive the observer. This prevents the implementation under test from quietly calling the evaluator’s verification path. In a real integration, the agent could perform a provider lookup through its own supported API, but the release gate should still retain an independent observation after the agent finishes.
4. Implement the independent observer
The observer reads provider state after the agent action. It does not parse the agent’s prose and does not infer state from the agent’s returned payment ID alone. It can search the provider’s read model using the request’s idempotency key in a richer implementation; this starter keeps the interface explicit and uses the agent’s ID only when one is available.
// src/observer.ts
import { AgentReport, Payment } from "./domain.js";
import { PaymentReadApi } from "./provider.js";
export interface Observation {
payment: Payment | undefined;
source: "provider-read-api";
}
export class IndependentObserver {
constructor(private readonly provider: PaymentReadApi) {}
async observe(report: AgentReport): Promise<Observation> {
const payment = report.paymentId
? await this.provider.getPayment(report.paymentId)
: undefined;
return {
payment,
source: "provider-read-api"
};
}
}
In production, this observer should use the strongest read path available: a provider retrieval endpoint, a reconciled database projection, or a signed event stream. The right choice depends on the system’s consistency model. The test requirement is stable: the evidence must come from an authoritative boundary that is not the agent’s own narration.
5. Create task fixtures
Fixtures make scenarios reviewable. Each fixture names the failure mode, the expected durable state, and whether a violation must block release.
// src/fixtures.ts
import { TaskFixture } from "./domain.js";
const baseRequest = {
amountMinor: 1250,
currency: "USD" as const,
idempotencyKey: "eval-order-001"
};
export const fixtures: TaskFixture[] = [
{
name: "provider accepts the payment",
request: baseRequest,
failureMode: "none",
critical: true,
expected: {
providerStatus: "created",
allowedClaimedStatuses: ["success"]
}
},
{
name: "provider rejects the payment",
request: baseRequest,
failureMode: "rejected",
critical: true,
expected: {
providerStatus: "absent",
allowedClaimedStatuses: ["failed", "unknown"]
}
},
{
name: "provider times out before commit",
request: baseRequest,
failureMode: "timeout-before-commit",
critical: true,
expected: {
providerStatus: "absent",
allowedClaimedStatuses: ["failed", "unknown"]
}
},
{
name: "provider times out after commit",
request: baseRequest,
failureMode: "timeout-after-commit",
critical: true,
expected: {
providerStatus: "absent",
allowedClaimedStatuses: ["pending", "unknown", "failed"]
}
},
{
name: "provider returns malformed success",
request: baseRequest,
failureMode: "malformed-response",
critical: true,
expected: {
providerStatus: "absent",
allowedClaimedStatuses: ["pending", "unknown", "failed"]
}
}
];
The expected observation for the timeout-after-commit fixture is marked as absent because the unsafe agent does not return the committed payment ID. The provider snapshot will still reveal that a record exists. That contradiction is useful: it demonstrates why the observer must not be restricted to the agent’s chosen identifier in a mature harness.
A production fixture should include a correlation key and an observer query that can find all records created by the scenario. For example, the fake could store the idempotency key on the payment and the observer could query by that key. Keep that extension explicit rather than pretending that an incomplete lookup is authoritative.
6. Validate both claims and state
The evaluator produces structured failures. A failure should state whether the problem was a claim mismatch, a state mismatch, or a missing observation. This makes a red gate actionable.
// src/evaluator.ts
import {
AgentReport,
Payment,
TaskFixture
} from "./domain.js";
import { Observation } from "./observer.js";
export interface EvaluationFailure {
code:
| "claim_not_allowed"
| "provider_state_mismatch"
| "success_without_observed_payment"
| "provider_record_not_reconciled";
message: string;
}
export interface Evaluation {
fixture: string;
critical: boolean;
passed: boolean;
failures: EvaluationFailure[];
report: AgentReport;
observation: Observation;
providerSnapshot: Payment[];
}
export function evaluate(
fixture: TaskFixture,
report: AgentReport,
observation: Observation,
providerSnapshot: Payment[]
): Evaluation {
const failures: EvaluationFailure[] = [];
if (!fixture.expected.allowedClaimedStatuses.includes(report.claimedStatus)) {
failures.push({
code: "claim_not_allowed",
message:
`Agent claimed ${report.claimedStatus}, but fixture allows ` +
`${fixture.expected.allowedClaimedStatuses.join(", ")}.`
});
}
const observedStatus = observation.payment?.status ?? "absent";
if (fixture.expected.providerStatus !== observedStatus) {
failures.push({
code: "provider_state_mismatch",
message:
`Expected observed provider state ${fixture.expected.providerStatus}, ` +
`got ${observedStatus}.`
});
}
if (
report.claimedStatus === "success" &&
!observation.payment
) {
failures.push({
code: "success_without_observed_payment",
message:
"Agent reported success but the independent observer found no payment."
});
}
if (
report.claimedStatus === "success" &&
providerSnapshot.length === 0
) {
failures.push({
code: "provider_record_not_reconciled",
message:
"Agent reported success but the provider snapshot is empty."
});
}
return {
fixture: fixture.name,
critical: fixture.critical,
passed: failures.length === 0,
failures,
report,
observation,
providerSnapshot
};
}
This evaluator intentionally treats a missing payment as different from an observed payment with a non-success status. That distinction matters when the provider has accepted a command but has not completed processing. “Not found,” “pending,” and “failed” are different facts and should not collapse into one convenient success or failure label.
7. Run each fixture through the harness
// src/harness.test.ts
import test from "node:test";
import assert from "node:assert/strict";
import { PaymentAgent } from "./agent.js";
import { evaluate } from "./evaluator.js";
import { fixtures } from "./fixtures.js";
import { IndependentObserver } from "./observer.js";
import { FakePaymentProvider } from "./provider.js";
for (const fixture of fixtures) {
test(fixture.name, async () => {
const provider = new FakePaymentProvider(fixture.failureMode);
const agent = new PaymentAgent(provider);
const observer = new IndependentObserver(provider);
const report = await agent.createPayment(fixture.request);
const observation = await observer.observe(report);
const result = evaluate(
fixture,
report,
observation,
provider.snapshot()
);
assert.equal(
result.passed,
true,
result.failures.map((failure) => failure.message).join("\n")
);
});
}
Run the tests with:
npm test
The unsafe agent is expected to fail at least the timeout-before-commit scenario because it reports success after catching an exception. It may also fail the malformed-response scenario because the response says “accepted” but contains no payment. These are expected outcomes of the example code, not test results verified in this article’s environment.
The article provides runnable code, but no command was executed as part of this publication. When repeating the work, record the Node.js version, TypeScript version, operating-system details, command, and output. That record is part of reproducibility.
8. Make the release gate critical
A test report is not a release policy. The gate must encode which failures are allowed to block a release. A critical mismatch between an agent claim and provider state should exit with a non-zero status even if other, non-critical checks pass.
// src/gate.ts
import { PaymentAgent } from "./agent.js";
import { evaluate } from "./evaluator.js";
import { fixtures } from "./fixtures.js";
import { IndependentObserver } from "./observer.js";
import { FakePaymentProvider } from "./provider.js";
const results = [];
for (const fixture of fixtures) {
const provider = new FakePaymentProvider(fixture.failureMode);
const agent = new PaymentAgent(provider);
const observer = new IndependentObserver(provider);
const report = await agent.createPayment(fixture.request);
const observation = await observer.observe(report);
results.push(
evaluate(
fixture,
report,
observation,
provider.snapshot()
)
);
}
for (const result of results) {
const prefix = result.passed ? "PASS" : "FAIL";
console.log(`${prefix} ${result.fixture}`);
for (const failure of result.failures) {
console.log(` - [${failure.code}] ${failure.message}`);
}
}
const criticalFailures = results.filter(
(result) => result.critical && !result.passed
);
if (criticalFailures.length > 0) {
console.error(
`Release blocked: ${criticalFailures.length} critical fixture(s) failed.`
);
process.exitCode = 1;
} else {
console.log("Release gate passed.");
}
Run the gate with:
npm run gate
In CI, the command should run before packaging, deployment, or promotion. Keep the policy visible in code and version it with the fixtures. If a team changes a fixture from critical to non-critical, that should be a deliberate review event rather than an incidental test edit.
Repairing the unsafe behavior
The smallest safe repair is to stop turning unknown outcomes into success. A stronger repair also performs reconciliation after ambiguous failures using the same idempotency key.
// The minimum safe behavior inside the catch branch:
return {
text: "The payment result is uncertain and requires verification.",
claimedStatus: "unknown"
};
That change prevents a false positive report, but it does not complete the business workflow. A useful production agent should expose uncertainty as a first-class result and route the case to a reconciliation step. The reconciliation step may query the provider by idempotency key, retrieve a known payment ID, or hand the case to an operator.
The release gate should test the complete contract:
- No provider rejection is reported as definite success.
- No timeout is reported as definite success without verification.
- A committed payment is not duplicated by a retry.
- The observed amount and currency match the requested amount and currency.
- The final message distinguishes created, pending, failed, and unknown states.
Strengthen state validation
Status alone is insufficient. A payment with the right status but the wrong amount is still a failed operation. Add exact checks for the fields that matter to the business action.
export function validatePayment(
payment: Payment | undefined,
request: CreatePaymentRequest
): string[] {
if (!payment) {
return ["payment_not_found"];
}
const failures: string[] = [];
if (payment.amountMinor !== request.amountMinor) {
failures.push("amount_mismatch");
}
if (payment.currency !== request.currency) {
failures.push("currency_mismatch");
}
return failures;
}
Apply this validation to the independent observation, not merely to the object returned by the command call. Otherwise the agent could pass a locally constructed object to the evaluator while the provider contains different durable state.
For systems with eventual consistency, make the observation policy explicit. A bounded retry can be valid when the provider documents delayed visibility:
type ObservePolicy = {
attempts: number;
delayMs: number;
};
Do not hide an unbounded wait inside the test. The fixture should record the policy and the evaluator should report whether the state was eventually observed, remained absent, or became contradictory. A timeout is evidence about the observation window, not proof that the provider never committed the action.
Failure cases worth keeping permanently
The following scenarios should remain in the suite because they protect against common regressions:
- Rejected request: the provider returns a structured rejection and no payment record. The agent must not say the payment succeeded.
- Timeout before commit: the provider throws before changing state. The agent must report uncertainty or failure, not success.
- Timeout after commit: the provider changes state and then throws. The agent must not blindly retry with a new idempotency key.
- Malformed accepted response: the provider indicates acceptance but omits the payment object. The agent must not manufacture a payment ID.
- Wrong amount: the provider record exists but does not match the request. The evaluator must block release.
- Wrong currency: the provider record exists with a different currency. The evaluator must block release.
- Duplicate retry: the same logical task is executed twice with the same idempotency key. The observer must find one logical payment, not two.
- Stale read: the first observation is empty and a later bounded observation finds the record. The result must be classified as eventually observed, not silently converted to success.
Keep at least one fixture for each distinct contradiction. A broad “provider error” fixture is not enough: it cannot tell reviewers whether the implementation handles rejection, pre-commit timeout, post-commit timeout, and malformed data differently.
What makes the eval reproducible
Reproducibility is a property of the whole experiment, not just of the test command. Record:
- The exact fixture identifier and version.
- The agent implementation revision.
- The provider failure mode.
- The observer policy and timeout window.
- The runtime and dependency versions.
- The structured report, observation, provider snapshot, and failure codes.
- The exit status of the release gate.
A machine-readable result makes this practical:
type GateRecord = {
fixture: string;
agentRevision: string;
providerFailure: ProviderFailure;
report: AgentReport;
observation: Observation;
failures: EvaluationFailure[];
releaseBlocked: boolean;
};
Do not include access tokens, live payment identifiers, customer names, card data, or copied production payloads in fixture output. Use synthetic identifiers and amounts. If a real incident motivates a fixture, reduce it to the smallest anonymized behavior that reproduces the defect.
How to interpret a red gate
A red gate is a release decision, not a diagnosis. Read the failure code and inspect the evidence in this order:
- What did the agent claim?
- What did the observer read?
- What records existed in the provider snapshot?
- Which fixture invariant failed?
- Was the observer’s read window appropriate for the provider’s consistency model?
If the agent claims success and the observer finds no record, the release should be blocked immediately. If the provider contains a record but the agent cannot identify it, investigate correlation and reconciliation. If the observer finds a pending record, do not label it succeeded merely because the command was accepted.
A useful failure message is specific enough to guide the next experiment:
FAIL provider times out before commit
- [claim_not_allowed] Agent claimed success, but fixture allows failed, unknown.
- [provider_state_mismatch] Expected observed provider state absent, got absent.
- [success_without_observed_payment] Agent reported success but the independent observer found no payment.
The duplicated “absent” line above illustrates a reporting detail worth refining: a production evaluator should avoid emitting redundant failures when one root cause explains several derived checks. Stable failure codes are more important than a long list of messages.
Limitations of this offline harness
This harness is intentionally bounded. It does not verify network transport, TLS behavior, provider authentication, webhook delivery, production database replication, rate limits, provider-specific status semantics, or the behavior of a real model. It also does not establish that a provider will behave exactly like the fake.
Its value is narrower and concrete: it verifies that the agent and evaluator distinguish a claimed outcome from observed external state for a defined set of failure modes. That is a necessary layer, not a complete production assurance program.
Add higher-level tests only when they answer a separate question. A contract test can check provider response schemas. An integration test can check authentication and real sandbox behavior. A replay test can check model determinism under a recorded prompt. A live canary can check operational behavior under controlled traffic. Keep each layer’s evidence separate so that a passing offline eval is not described as proof of production correctness.
A practical review checklist
Before approving an agent that changes external state, ask:
- Is there a fixture for a provider rejection?
- Is there a fixture for timeout before commit?
- Is there a fixture for timeout after commit?
- Can the observer read state without trusting the agent’s prose?
- Are amount, currency, ownership, and status validated?
- Does the agent represent unknown outcomes explicitly?
- Is idempotency tested across retries?
- Do critical mismatches produce a non-zero process exit code?
- Are test outputs free of secrets and personal data?
- Can another engineer run the same command from a clean checkout?
If the answer to the observer question is no, the system is still testing the agent’s story about the action rather than the action itself.
Conclusion
The essential pattern is straightforward: give the agent a controlled task, let it call a deterministic external-system double, read the resulting state through an independent observer, and block release when the claim and the evidence disagree. The hard part is resisting the convenience of treating a successful function return or a confident sentence as proof.
Start with a small offline harness. Add one fixture for every meaningful ambiguity. Keep failure injection deterministic. Validate durable state rather than conversational intent. Then make the critical evaluator part of the release command, so that a known false-success path cannot quietly become a deployment detail.
Continue with the Agent Lab Journal guides or look up related terms in the glossary.