Reproducibility · Advanced · 25 min read

How to Reproduce an Agent-Memory Benchmark and Audit Its Submission Format

A benchmark result is only as useful as the path from input to published number. This guide shows how to rebuild that path, inspect what the agent actually saw, and reject an archive that cannot be independently audited.

By Agent Lab Journal Updated 6 November 2026 25 min read

Comparing agent-memory systems looks straightforward: run the same tasks, collect the scores, and compare the averages. In practice, the comparison can change before the memory component is even evaluated. A reference runner may alter prompts, a result format may omit failed episodes, and a publication process may expose only a final table rather than the evidence needed to reconstruct it.

The reliable unit of comparison is therefore not a number. It is a traceable experiment: a pinned runner, a declared configuration, immutable task inputs, complete transcripts, explicit metrics, and a submission archive that another person can inspect without private context.

A concrete audit case

Imagine a benchmark named DolphinBench-Memory. It evaluates whether an agent can answer questions about information introduced earlier in a multi-turn episode. The benchmark repository contains a runner, task files, a model adapter, and a submission template. A paper reports one aggregate score, but the accompanying archive contains only a CSV with one row per system.

That archive is not automatically invalid. It is simply insufficient until you can answer questions such as:

• Which exact task instances were evaluated?
• Did every system receive the same history and tool results?
• Were retries, timeouts, and malformed outputs counted?
• Was the score computed from all episodes or only successful episodes?
• Can a reviewer map every aggregate value back to an episode-level record?

Your goal is to turn those questions into checks with observable evidence. If a property cannot be checked from the runner, configuration, logs, or archive, label it unverified rather than inferring it from a polished table.

Model the experiment before running it

Write down the experiment as a function before installing dependencies:

score = evaluate(
    runner_version,
    task_set,
    agent_config,
    model_identifier,
    seed_policy,
    environment,
    scoring_rules
)

Each argument can affect the result. Record at least:

This inventory is also where you separate verified facts from examples. A command such as git rev-parse HEAD can verify a local commit. A statement such as “the benchmark uses deterministic evaluation” requires evidence about the runner, model API, sampling parameters, and external tools.

Create a controlled environment

Use an isolated environment and make the dependency state visible. The commands below are illustrative; they are not claims about a particular repository.

git clone <verified-benchmark-repository>
cd <repository-directory>

git checkout --detach <verified-commit>

python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

python --version
python -m pip freeze > reproduction/pip-freeze.txt
git rev-parse HEAD > reproduction/runner-commit.txt

Do not place API keys in shell history, configuration committed to the archive, or transcripts. Load secrets from the execution environment and redact them before publication:

export BENCHMARK_API_KEY='<provided-out-of-band>'
python -m benchmark_runner \
  --config configs/reproduction.yaml \
  --output runs/example-001

Keep the original configuration beside the run. If the runner expands defaults, save the resolved configuration too. A short configuration might look like this:

tasks_manifest: data/tasks.sha256
episodes: all
seed: 17
temperature: 0
max_retries: 0
timeout_seconds: 120
save_transcripts: true
save_tool_events: true
score_from: episode_results.jsonl

“Temperature zero” is not a proof of identical outputs. Provider-side changes, tool timing, hidden sampling, and retrieval order can still introduce variation. Treat determinism as a hypothesis to test.

Run a smoke test, then the full benchmark

Start with one known task and one deliberately failing task if the runner supports them. The smoke test should prove that the pipeline creates the expected artifacts and that a failure remains visible.

python -m benchmark_runner \
  --config configs/reproduction.yaml \
  --task-id smoke-001 \
  --output runs/smoke-001

find runs/smoke-001 -maxdepth 2 -type f -print | sort

Check for a run manifest, resolved configuration, episode results, transcript files, error records, and a machine-readable summary. If the smoke test silently drops an exception, stop there. A full run built on an opaque smoke test is not reproducible.

For the full run, preserve the command line exactly. A shell script or task runner is preferable because it records the invocation and makes accidental changes easier to detect:

set -eu

RUN_ID="system-a-seed-17"
python -m benchmark_runner \
  --config configs/reproduction.yaml \
  --system system-a \
  --seed 17 \
  --output "runs/${RUN_ID}"

python -m benchmark_tools.validate_run \
  --run "runs/${RUN_ID}" \
  --manifest data/tasks.sha256

Run each system against the same task manifest. Do not regenerate tasks separately for each system unless the benchmark explicitly defines that procedure. Preserve ordering even when the scoring formula is order-independent; ordering often reveals missing or duplicated episodes.

Audit transcripts, not just scores

A transcript is evidence of what the agent received and produced. It should make the memory pathway inspectable without exposing secrets or personal data. At minimum, each episode should identify:

• task ID and run ID;
• runner and task-set versions;
• turn order and message roles;
• memory writes, reads, or retrieval events;
• tool calls and tool results;
• termination reason;
• episode-level score and error status.

Use a schema validator before manual review:

python -m benchmark_tools.validate_transcripts \
  --input runs/system-a-seed-17/transcripts \
  --schema schemas/transcript.schema.json

Then inspect representative episodes: a correct answer requiring recall, a correct answer that does not require memory, a memory miss, a timeout, and a malformed response. These examples expose different classes of failure. A transcript that contains only the final answer cannot show whether the agent remembered, guessed, or received leaked context.

Redaction must preserve auditability. Replace secrets with stable placeholders such as [REDACTED_API_KEY_1], not arbitrary deletions. If personal data appears in a task, remove or synthesize it before publication and document that transformation.

Calculate metrics from episode-level records

Do not accept a reported aggregate until you can reproduce it from the lowest useful level of data. A simple accuracy definition is:

accuracy = sum(episode.correct for episode in episodes) / len(episodes)

Real benchmarks often add partial credit, abstention handling, latency, cost, or memory-specific measures. Define the denominator for every metric. For example, “accuracy on completed episodes” and “accuracy over all assigned episodes” are different quantities when timeouts exist.

A compact validation script can compare the archive’s summary with a recomputation:

python -m benchmark_tools.recompute_metrics \
  --episodes runs/system-a-seed-17/episode_results.jsonl \
  --output runs/system-a-seed-17/recomputed_metrics.json

diff -u \
  runs/system-a-seed-17/summary.json \
  runs/system-a-seed-17/recomputed_metrics.json

Report counts alongside rates:

• assigned episodes;
• completed episodes;
• correct, incorrect, abstained, and errored episodes;
• timeouts and retries;
• episodes excluded by policy;
• the exact denominator used.

If repeated runs are possible, report variation rather than a single lucky score. Keep the seed policy fixed and distinguish repeated seeds from repeated provider calls. A confidence interval can describe sampling uncertainty, but it does not repair systematic differences in prompts, task ordering, or runner behavior.

Validate the submission archive

The submission archive should be self-describing. A practical layout is:

submission/
├── README.md
├── manifest.sha256
├── metadata.json
├── configs/
│   └── resolved.yaml
├── runs/
│   └── system-a-seed-17/
│       ├── run.json
│       ├── episode_results.jsonl
│       ├── metrics.json
│       ├── transcripts/
│       └── errors.jsonl
└── schemas/
    └── transcript.schema.json

The submission archive must answer four questions:

1. What was run? Include versions, configuration, command, environment, and task manifest.
2. What happened? Include episode records, transcripts, tool events, and errors.
3. How was it scored? Include formulas, denominators, exclusions, and recomputed outputs.
4. Can files be verified? Include hashes and a documented validation command.

Create hashes after final redaction and packaging:

find submission -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum \
  > submission/manifest.sha256

sha256sum --check submission/manifest.sha256

Check the archive in a clean directory, not only in the working tree where untracked files may hide dependencies:

tmpdir="$(mktemp -d)"
tar -xzf submission.tar.gz -C "$tmpdir"

python -m benchmark_tools.validate_submission \
  --root "$tmpdir/submission"

python -m benchmark_tools.recompute_metrics \
  --episodes "$tmpdir/submission/runs/system-a-seed-17/episode_results.jsonl" \
  --output "$tmpdir/recomputed.json"

Only call the archive reproducible if the clean-room validation succeeds and the recomputed metrics match under a declared tolerance. If the archive depends on an unavailable private endpoint, mark the run as not independently rerunnable and explain what evidence remains available.

Common failure cases

Different prompts hidden behind the same system name

Two agents may both be called “memory-enabled” while one receives retrieved memories in a system message and the other receives them as user content. Compare resolved prompts and event logs, not labels.

Retries counted inconsistently

A retry can be a new attempt, a continuation of the same episode, or a failure that should count against the system. Declare the policy and preserve every attempt in the raw record.

Missing episodes replaced by a smaller denominator

If a runner writes only successful outputs, aggregate accuracy can be inflated. Reconcile the task manifest with the result IDs and require an explicit status for every assigned episode.

Tool results are not captured

Memory behavior cannot be audited if external retrieval, search, or database responses are absent from the transcript. Save tool inputs and outputs, subject to redaction.

Archive validates only on the author’s machine

Undeclared local packages, environment variables, mounted files, and cached model assets are common causes. Run the validation from a clean environment and list unavoidable external dependencies.

What this procedure cannot establish

A successful reproduction establishes that a declared pipeline can produce a comparable run. It does not prove that the benchmark measures long-term memory in every setting, that the task set is free from leakage, or that one score represents all agent workloads.

It also cannot remove changes outside the archive’s control, such as an updated model service or unavailable historical dependency. The correct response is to record the uncertainty, narrow the claim, and preserve enough intermediate evidence for a later audit.

Final reproducibility checklist

• Runner commit and task manifest are pinned.
• Resolved configuration is included.
• Model identifier and sampling policy are recorded.
• Every assigned episode has a terminal status.
• Transcripts include memory and tool events.
• Secrets and personal data are removed or transformed.
• Metrics are recomputable from episode-level records.
• Counts and denominators accompany aggregate rates.
• Archive hashes are generated after redaction.
• Validation succeeds in a clean directory.
• Unverified assumptions are labeled as limitations.

The central discipline is simple: follow the evidence from task manifest to transcript to episode result to aggregate table. When a link is missing, do not fill it with confidence. Mark the gap, preserve the available artifacts, and state exactly what a future reader can and cannot reproduce.

For related procedures and terminology, see the guides and the glossary.