Advanced · 10-minute read · Reproducible evaluation plan

How to Compare AI Agents on Wet-Lab Data Interpretation

Как сравнить AI-агентов на задачах интерпретации лабораторных данных
Temporary fallback cover; replace in editorial pass.

A general model benchmark can show that one system answers more questions correctly than another. It cannot show whether an agent can inspect a plate map, detect a failed control, preserve sample identity, explain an unexpected curve, and stop before making an unsupported biological claim. For that, you need a task-level evaluation built around real experimental-data workflows.

This article describes a comparison plan for 100 wet-lab tasks. It is a protocol for building and auditing the evaluation, not a report of test results. No agent scores, laboratory findings, or performance claims are asserted here.

What the comparison should measure

The unit of evaluation is an AI agent run on one bounded task, with access to the same files, instructions, tools, and time budget as every other agent. A task should require interpretation of experimental evidence, not merely recall of textbook facts.

The comparison should measure five separate abilities:

These dimensions should be scored separately. A single pass/fail label hides useful failure patterns: an agent may calculate correctly while silently swapping sample labels, or produce a reasonable interpretation while using an invalid denominator.

Build the 100-task set

Start with a task matrix before collecting data. The matrix prevents the benchmark from becoming a collection of convenient examples selected after seeing which agent performs well.

Task family Suggested count Typical evidence
Quality control and integrity 20 Controls, missing values, outliers, plate layouts, instrument flags
Quantification 20 Standard curves, Ct values, absorbance, fluorescence, calibration files
Comparative analysis 20 Conditions, replicates, dose responses, time courses
Image or trace interpretation 15 Microscopy summaries, gels, chromatograms, growth curves
Cross-file reconciliation 15 Metadata, sample sheets, instrument exports, protocol notes
Ambiguous or non-conclusive cases 10 Conflicting controls, insufficient replicates, incomplete metadata

The counts above are a planning example. Replace them with a documented sampling rule that reflects the laboratory workflows you want to compare. Record why each task belongs in the set and which capability it tests.

Each task should contain a stable identifier, a short user request, the input files, allowed tools, expected output format, known hazards, and an adjudicated answer key. Keep the answer key separate from the agent prompt. If the agent can access it directly or indirectly, the task has data leakage.

Define the task contract

A reproducible task contract makes runs comparable. Use a machine-readable record such as:

{
  "task_id": "qcpcr_017",
  "request": "Assess whether the treatment effect is interpretable.",
  "inputs": [
    "plate_readout.csv",
    "sample_sheet.csv",
    "protocol_notes.md"
  ],
  "allowed_tools": ["python", "local_files"],
  "output_schema": {
    "observations": "array",
    "analysis": "object",
    "conclusion": "string",
    "limitations": "array",
    "artifacts": "array"
  },
  "time_limit_minutes": 20
}

Use the same prompt wrapper, file names, tool permissions, temperature policy, and maximum number of retries for every agent. If an agent has a different interface, document the adapter rather than quietly changing the task.

Ask for both a human-readable report and structured fields. The report helps reviewers understand reasoning; the structured fields make automated checks possible. Require the agent to list observations before conclusions, cite the input artifact used for each key claim, and state what additional evidence would change its conclusion.

Construct answer keys without pretending there is one truth

Wet-lab interpretation often has a verifiable core and a judgment-dependent edge. The answer key should represent both.

For every task, identify:

Use two independent reviewers for the answer key where possible. Resolve disagreements in a recorded adjudication note. Reviewers should not see agent identity while scoring. This reduces, but does not eliminate, subjective variation.

Score correctness and safety separately

A practical rubric uses a 0–3 scale for each dimension:

Score at least these dimensions independently: sample and metadata integrity, numerical correctness, method suitability, interpretation, uncertainty, provenance, and output completeness. Report the distribution of scores, not only the average.

Add critical-error flags that override a high average. Examples include swapping treatment and control, treating a failed control as valid, inventing a missing measurement, reporting a fabricated file or tool call, or presenting a biological conclusion when the task is explicitly inconclusive.

Run the evaluation reproducibly

Store one immutable run directory per agent and task. A useful layout is:

runs/
  agent-a/
    qcpcr_017/
      task.json
      prompt.txt
      inputs.sha256
      response.md
      structured.json
      tool_log.jsonl
      artifacts/
      evaluator_scores.json
      adjudication.md

Hash the input files before each run and record the agent version, model identifier, configuration, start and end times, tool versions, and failure status. Keep secrets out of prompts, logs, and artifacts. Use synthetic or properly de-identified data unless the governance process explicitly permits another dataset.

A minimal integrity check can be performed with:

find inputs -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum > inputs.sha256

sha256sum --check inputs.sha256

The commands above are examples of a reproducibility procedure; they have not been run as part of this article. Adapt them to your operating system and storage policy.

Verify before scoring

Verification should occur in two passes. First, run deterministic checks: required fields exist, numerical outputs parse, units are present, referenced files exist, and reported values fall within documented tolerances. Second, perform blinded human review for method choice, interpretation, caveats, and unsupported claims.

Do not treat a fluent explanation as evidence of a correct analysis. Recalculate important values from the original inputs. Open generated plots and inspect their labels, scales, legends, and grouping. Check whether the agent used all relevant replicates and whether exclusions were justified.

Report inter-reviewer agreement when more than one reviewer scores a task. If reviewers disagree, retain both initial scores and the adjudicated score. The disagreement itself may reveal that the task or rubric is underspecified.

Perform error analysis, not just ranking

After scoring, group failures by mechanism rather than by agent name alone. Useful error categories include:

For each failure, preserve the smallest evidence bundle needed to reproduce it: the task identifier, input hash, relevant prompt segment, response excerpt, tool trace, expected behavior, and reviewer rationale. Count both task-level failures and error instances, because one response can contain multiple independent errors.

Limitations and failure cases

One hundred tasks can provide a useful comparison without representing every organism, assay, instrument, or laboratory. Results will be sensitive to task construction, file cleanliness, prompt wording, reviewer judgment, and tool access. A benchmark can also reward the conventions encoded in its answer keys.

Avoid changing the task set after seeing results unless you label the new version and rerun every agent. Do not mix exploratory and confirmatory tasks in one score. Do not compare runs made with different input files, hidden context, retry budgets, or model versions without making those differences explicit.

Most importantly, an agent that refuses an unsupported conclusion may be performing better than an agent that produces a confident answer. The evaluation must reward justified uncertainty, not merely completion.

Minimum reproducible report

Publish or archive the task manifest, prompt wrapper, input hashes, agent configurations, raw outputs, structured outputs, scoring rubric, reviewer instructions, exclusions, and error taxonomy. Report coverage, missing runs, critical-error counts, per-dimension score distributions, and representative failure cases. If the data cannot be shared, provide a synthetic replacement or a controlled-access procedure and explain what cannot be independently verified.

This produces a defensible comparison: readers can see what was tested, what was observed, how judgments were made, and where the evidence stops. That is more informative than a single leaderboard number for deciding whether an agent is ready for a particular wet-lab workflow.

Continue with the Agent Lab Journal guides or check the glossary for the evaluation terms used here.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →