Самоулучшение coding-агента: одна ошибка против архива запусков

Самоулучшение coding-агента: одна ошибка против архива запусков
Temporary fallback cover; replace in editorial pass.

Level: advanced · Reading and practice time: about 90 minutes · Updated:

Your agent fails a task. You open the log, find the step where it went wrong, and add a line to the system prompt: "always run the tests before you finish". On the next run that task passes. Most likely you have fixed nothing. A single failed trajectory gives you one story, and a story is not a cause. This article gives you a three-arm experiment you can run on your own coding agent. Arm A mutates the agent from one failure. Arm B mutates it from an archive of many runs. Arm C does the same as B and adds the trajectory of a competing agent that solved the same tasks. All three arms are scored on tasks the mutation process never saw.

1. Why one failure misleads

A coding agent is a model running in a loop. It reads a repository, makes tool calls such as reading files, editing code and running commands, and stops when it decides the task is finished. Each run leaves a trajectory: the ordered list of messages, tool calls, tool outputs and the final diff.

When people talk about self-improvement of an agent, they usually mean a loop like this. Run the agent, look at what went wrong, change the agent, and run it again. The "change" is a mutation. It can be an edit to the system prompt, a new tool, a changed tool description, a new step in the scaffold, or a different stop condition. Sometimes a human writes the mutation. Sometimes the agent or a separate model proposes it.

If the mutation is built from one failed trajectory, three things work against you:

An archive of runs addresses all three. Repeated runs show how often a task fails. Many failures let you group symptoms and look for causes they share. A held-out set tells you whether a mutation generalises. A competing agent's trajectory adds one more thing: an example of a different route to success on the same task. That shows you what your agent did not do. Its own failures cannot show you that.

That is the claim. The rest of the article is a way to test it on your own system rather than take it on trust.

2. An illustrative case

This case is a constructed example. It shows the kind of reasoning involved. It is not a record of a real run.

Take an agent working on a medium-sized Python service. The task is "the /export endpoint returns dates in the wrong time zone; fix it and add a test". In one run the agent does the following:

  1. It searches for export and opens api/export.py.
  2. It sees a call to format_date() and edits the format string in place.
  3. It writes a test that calls format_date() directly. The test passes.
  4. It finishes. The hidden acceptance test, which calls the HTTP endpoint, fails because the time zone is applied in the serializer, not in format_date().

From this one trajectory the obvious mutation is: "Before you finish, run the full test suite." That rule would not have helped. The agent's own test passed, and the acceptance test was hidden.

Now suppose you have an archive. It holds forty tasks with five runs each, and failures are labelled. You might find a cluster in which, across many tasks, the agent edits the first function that matches a search and never traces the call path to the place the value is actually produced. In that cluster the "tests" symptom is incidental. The shared cause is shallow localisation. The mutation that follows is different: "Before editing, trace the value from the entry point to where it is produced and name the file in which you will make the change."

Suppose a competing agent solved the same task. Its trajectory shows it ran the endpoint with a request first, saw the wrong output, and then searched for the serializer. That suggests a third, more concrete mutation: "Reproduce the bug through the public interface before you read the code."

Which of the three mutations is actually better is an empirical question. The rest of the article answers it with an experiment.

3. Experiment design: three arms

All three arms start from the same baseline agent: the same model, scaffold, tools and limits. They differ only in what the mutation proposer is allowed to see.

ArmInput to the proposerWhat it simulates
A. Single failureOne failed trajectory, picked at random from the training splitThe usual practice: "it broke, patch the prompt"
B. ArchiveFailure clusters from the whole training archive, with 2–3 example trajectories per cluster and per-task failure ratesA diagnosis that looks for causes across many runs
C. Archive + competitorEverything in B, plus successful trajectories from a competing agent on the same tasks where the baseline failedA diagnosis that also learns from another agent's route to success
0. ControlNo mutation. The baseline is re-run with the same budgetRun-to-run noise and drift in the environment

Fix these rules before you start and write them down in a PROTOCOL.md:

Your hypotheses, stated so the data can contradict them:

H0 is a legitimate outcome. If you get it, that is a finding about your agent and your tasks. It is not a failed experiment.

4. Step 1. Tasks, splits and harness

4.1. The task set

You need tasks with an automatic and trustworthy acceptance check. The most practical source is your own repositories: take closed bug reports with a fixing commit, roll the repository back to the parent commit, and use the tests from the fix as the hidden check. A public benchmark of the "repository + issue + hidden tests" kind also works, if its licence allows it.

Requirements:

4.2. Splits

Split the tasks once, before any agent runs, and stratify by category:

# split_tasks.py — fixed, stratified split
import json, random, collections, sys

SEED = 20261017
tasks = [json.loads(l) for l in open("tasks.jsonl")]
by_cat = collections.defaultdict(list)
for t in tasks:
    by_cat[t["category"]].append(t)

rng = random.Random(SEED)
split = {"train": [], "val": [], "test": []}
for cat, items in sorted(by_cat.items()):
    rng.shuffle(items)
    n = len(items)
    a, b = n // 2, n // 2 + n // 4
    split["train"] += items[:a]
    split["val"]   += items[a:b]
    split["test"]  += items[b:]

for name, items in split.items():
    with open(f"split_{name}.jsonl", "w") as f:
        for t in items:
            f.write(json.dumps(t, ensure_ascii=False) + "\n")
    print(name, len(items), file=sys.stderr)

Commit the split files and the seed. If you ever re-split "because the test split came out too hard", the experiment is no longer valid.

4.3. The harness

The harness is the code that starts the agent on a task in an isolated environment, applies the limits, saves the trajectory and runs the hidden check. The requirements are what matter here, not any particular framework:

An example configuration layout:

# agent/baseline.yaml
id: baseline
model: your-model-id          # the same in every arm
temperature: 0.7              # leave it non-zero, or there is no variance to measure
max_steps: 60
max_wall_seconds: 900
tools:
  - read_file
  - search
  - edit_file
  - run_command
system_prompt_file: prompts/baseline.md
tool_descriptions_file: prompts/tools.md

The run driver. Replace run-agent with your own entry point:

#!/usr/bin/env bash
# run_batch.sh CONFIG SPLIT REPEATS OUTDIR
set -euo pipefail
CONFIG="$1"; SPLIT="$2"; REPEATS="$3"; OUT="$4"
mkdir -p "$OUT"
CFG_HASH=$(cat "$CONFIG" prompts/*.md | sha256sum | cut -c1-12)

jq -c '.' "split_${SPLIT}.jsonl" | while read -r task; do
  TID=$(echo "$task" | jq -r .id)
  for r in $(seq 1 "$REPEATS"); do
    RUN_ID="${TID}__${CFG_HASH}__r${r}"
    [ -f "$OUT/$RUN_ID.json" ] && continue      # resume safely after a crash
    run-agent --config "$CONFIG" --task-json "$task" \
              --trajectory-out "$OUT/$RUN_ID.traj.jsonl" \
              --result-out "$OUT/$RUN_ID.json" \
              --seed "$r" || echo "{\"run_id\":\"$RUN_ID\",\"status\":\"harness_error\"}" > "$OUT/$RUN_ID.json"
  done
done

Keep harness_error separate from an ordinary failure. A broken container is not a mistake by the agent, and it must never get into the archive as one.

5. Step 2. Build a run archive

Run the baseline on train with at least 5 repeats per task, and on val and test with the same number of repeats. You need the baseline numbers on val and test for comparison later. The proposer must not see them.

./run_batch.sh agent/baseline.yaml train 5 runs/baseline/train
./run_batch.sh agent/baseline.yaml val   5 runs/baseline/val
./run_batch.sh agent/baseline.yaml test  5 runs/baseline/test

The result record for one run:

{
  "run_id": "exp-017__a1b2c3d4e5f6__r3",
  "task_id": "exp-017",
  "category": "localisation",
  "config_hash": "a1b2c3d4e5f6",
  "status": "fail",
  "steps": 41,
  "wall_seconds": 512,
  "hidden_tests": {"passed": 2, "failed": 1},
  "files_edited": ["api/export.py", "tests/test_export.py"],
  "stop_reason": "agent_declared_done"
}

Now build a per-task table. This is the first thing a single trajectory cannot give you:

# task_rates.py — pass rate per task across repeats
import json, glob, collections

rows = collections.defaultdict(lambda: {"pass": 0, "fail": 0, "error": 0})
for p in glob.glob("runs/baseline/train/*.json"):
    r = json.load(open(p))
    key = {"pass": "pass", "fail": "fail"}.get(r["status"], "error")
    rows[r["task_id"]][key] += 1

print("task_id\tpass\tfail\terror\tclass")
for tid, c in sorted(rows.items()):
    n = c["pass"] + c["fail"]
    rate = c["pass"] / n if n else float("nan")
    cls = ("always_pass" if rate == 1 else
           "always_fail" if rate == 0 else "unstable")
    print(f"{tid}\t{c['pass']}\t{c['fail']}\t{c['error']}\t{cls}")

The three classes call for different action:

5.1. The competitor's trajectories (Arm C only)

Run the competing agent on the same train split with the same harness, limits and number of repeats. The competitor can be a different scaffold, a different model, or a stronger configuration of your own agent. Keep only successful competitor runs on tasks where the baseline is always_fail or unstable. Those are the tasks where the comparison carries information.

./run_batch.sh agent/competitor.yaml train 5 runs/competitor/train

Check the competitor's terms of use before you put its trajectories into another system's prompt. If the competitor is a third-party product, its outputs may come with restrictions. Do not include customer code or secrets from the trajectories in anything you publish.

6. Step 3. Turn failures into labelled clusters

Raw trajectories are too long and too many to give to the proposer. Arm B needs a summary: which kinds of failure occur, how often, and on which tasks. Do this in two passes. The first is automatic, and in the second a human checks the result.

6.1. Automatic first pass: one failure, one record

For each failed run, ask a model, separate from the agent and the proposer, to fill a fixed form. A fixed form matters. Free-text "post-mortems" are hard to group.

You are reviewing one failed run of a coding agent.
You get: the task, the full trajectory, the final diff and the hidden test output.
Do not propose fixes. Fill the form below, nothing else. Output JSON only.

{
  "first_wrong_step": <step index where the run first went off track>,
  "symptom": "<what visibly failed, one sentence>",
  "root_cause_category": one of [
      "shallow_localisation", "misread_task", "no_reproduction",
      "wrong_tool_use", "premature_stop", "env_or_dependency",
      "test_written_to_pass", "context_loss", "other"],
  "evidence_steps": [<step indices that support the category>],
  "confidence": "low" | "medium" | "high"
}

The list of categories is an example. Adjust it to what your agent actually gets wrong, but freeze it before the arms start.

6.2. Human check

Take a random sample of about 30 labelled failures, or all of them if there are fewer. Check each label against the trajectory. Record whether you agree. If you disagree on more than about a third, revise the category definitions and relabel. Do not hand clusters you do not believe in to the proposer.

This step is slow. It is also the most valuable part of the experiment. It is often where you discover the real problem, before any mutation has been tried.

6.3. Build the evidence block

# build_evidence.py — evidence block for Arm B
import json, glob, collections, random

labels = [json.loads(l) for l in open("labels_train.jsonl")]   # one per failed run
rates  = {r["task_id"]: r for r in map(json.loads, open("task_rates.jsonl"))}

clusters = collections.defaultdict(list)
for lab in labels:
    clusters[lab["root_cause_category"]].append(lab)

rng = random.Random(7)
out = []
for cat, items in sorted(clusters.items(), key=lambda kv: -len(kv[1])):
    tasks = sorted({i["task_id"] for i in items})
    examples = rng.sample(items, min(3, len(items)))
    out.append({
        "category": cat,
        "failed_runs": len(items),
        "distinct_tasks": len(tasks),
        "always_fail_tasks": sum(rates[t]["class"] == "always_fail" for t in tasks),
        "example_excerpts": [
            {"task_id": e["task_id"],
             "symptom": e["symptom"],
             "excerpt": e["excerpt"]}        # steps around first_wrong_step, max ~60 lines
            for e in examples],
    })
json.dump(out, open("evidence_B.json", "w"), ensure_ascii=False, indent=2)

Count distinct_tasks as well as runs. A cluster of 15 failed runs that all come from one task is a problem with one task, not a pattern.

For Arm C, add the competitor's trajectories to each cluster as excerpts: the corresponding stretch of a successful run on a task from that cluster. Align them by stage: how the agent found the place to change, how it checked the change, how it decided to stop. Do not paste whole trajectories in.

6.4. The Arm A input

For Arm A, pick one failed run at random from the train split, using a recorded seed. Give the proposer that run's excerpt and symptom, without the cluster statistics. Make five draws with different seeds. Each draw is a separate instance of Arm A.

# pick_single.py SEED
import json, random, sys
labels = [json.loads(l) for l in open("labels_train.jsonl")]
rng = random.Random(int(sys.argv[1]))
choice = rng.choice(labels)
json.dump({"single_failure": choice}, open(f"evidence_A_seed{sys.argv[1]}.json", "w"),
          ensure_ascii=False, indent=2)

7. Step 4. Generate mutations for each arm

One proposer template for every arm. Only {EVIDENCE} changes:

You improve a coding agent by editing its system prompt.

Current system prompt:
<<<
{CURRENT_PROMPT}
>>>

Evidence about its failures:
<<<
{EVIDENCE}
>>>

Rules:
- Propose exactly {K} alternative edits. Each edit is independent of the others.
- Each edit adds, removes or rewrites at most 15 lines.
- Do not mention specific file names, function names or task ids from the evidence.
  The edit must apply to unseen repositories.
- For each edit, state the failure mechanism it targets and how you would
  recognise in a trajectory that it worked.

Output JSON: [{"id": "...", "diff": "<unified diff of the prompt>",
               "mechanism": "...", "observable_signal": "..."}]

The rule against task-specific names matters most for Arm A. Without it the proposer will often write a rule that only fits the one failure it saw. Enforce the rule with a check, not only with the prompt:

# lint_mutation.py — reject mutations that leak task specifics
import json, re, sys
mut = json.load(open(sys.argv[1]))
leak_terms = set()
for t in map(json.loads, open("split_train.jsonl")):
    leak_terms |= set(t.get("identifiers", []))      # file/function names you listed per task
for m in mut:
    added = "\n".join(l[1:] for l in m["diff"].splitlines() if l.startswith("+"))
    n_lines = sum(1 for l in m["diff"].splitlines() if l[:1] in "+-" and not l.startswith(("+++", "---")))
    hits = [w for w in leak_terms if re.search(rf"\b{re.escape(w)}\b", added)]
    status = "ok" if not hits and n_lines <= 15 else "reject"
    print(m["id"], status, n_lines, hits)

If a mutation is rejected, ask the proposer for a replacement, so that each arm still ends up with K valid mutations. Record how many were rejected in each arm. That count is a result in its own right.

Apply each mutation to a copy of the configuration:

# for each arm and each mutation id
cp -r agent agent_variants/B_m3
patch agent_variants/B_m3/prompts/baseline.md < mutations/B_m3.diff
sed -i 's/^id: baseline/id: B_m3/' agent_variants/B_m3/baseline.yaml

8. Step 5. Evaluate on held-out tasks

8.1. Selection on val

Run every mutation in every arm on the val split with 3–5 repeats. In each arm, pick the mutation with the highest mean pass rate. Break ties by fewer regressions on tasks that were always_pass for the baseline.

for v in agent_variants/*/; do
  name=$(basename "$v")
  ./run_batch.sh "$v/baseline.yaml" val 3 "runs/$name/val"
done

Selection is itself a small optimisation over val, so a winner's val score is biased upwards. That is why there is a test split.

8.2. The final run on test

Run only the winners on test (one winner each for B and C, and one winner per seed for A), together with the control (the baseline again), with 5 repeats each. Run all of them in the same time window. API models and environments change, and the baseline from a week ago is not a fair control.

for name in control A_s1_win A_s2_win A_s3_win A_s4_win A_s5_win B_win C_win; do
  ./run_batch.sh "agent_variants/$name/baseline.yaml" test 5 "runs/$name/test"
done

8.3. What to record besides pass rate

# signal_check.py — did the targeted behaviour appear?
import json, glob, re, sys
variant, pattern = sys.argv[1], sys.argv[2]   # e.g. B_win "run_command.*(curl|pytest -k|python -c)"
hits = total = 0
for p in glob.glob(f"runs/{variant}/test/*.traj.jsonl"):
    steps = [json.loads(l) for l in open(p)]
    first_edit = next((i for i, s in enumerate(steps) if s.get("tool") == "edit_file"), len(steps))
    before = steps[:first_edit]
    total += 1
    hits += any(s.get("tool") == "run_command" and re.search(pattern, s.get("input", "")) for s in before)
print(f"{variant}: signal before first edit in {hits}/{total} runs")

Run the same check on the control. If the baseline already does the same thing just as often, the mutation did not change the behaviour it claims to change.

9. Step 6. Compare arms honestly

The unit of analysis is the task, not the run. Runs of the same task are not independent. For each variant and task, compute the mean pass rate across the 5 repeats. Then compare variants on paired task-level differences.

# compare.py — paired bootstrap over tasks
import json, glob, random, statistics, collections, sys

def task_means(variant):
    acc = collections.defaultdict(list)
    for p in glob.glob(f"runs/{variant}/test/*.json"):
        r = json.load(open(p))
        if r["status"] in ("pass", "fail"):
            acc[r["task_id"]].append(1.0 if r["status"] == "pass" else 0.0)
    return {t: sum(v) / len(v) for t, v in acc.items()}

a, b = sys.argv[1], sys.argv[2]
ma, mb = task_means(a), task_means(b)
tasks = sorted(set(ma) & set(mb))
diffs = [mb[t] - ma[t] for t in tasks]

rng = random.Random(1)
boots = []
for _ in range(10000):
    sample = [diffs[rng.randrange(len(diffs))] for _ in diffs]
    boots.append(sum(sample) / len(sample))
boots.sort()
lo, hi = boots[249], boots[9749]
print(f"{b} - {a}: mean diff {statistics.mean(diffs):+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}], n_tasks={len(tasks)}")

This is a bootstrap resampling of tasks. If the interval includes zero, the data do not show a difference at this sample size. Write it that way, without "trends".

The comparisons to make:

  1. Each arm against the control. Is there any effect at all?
  2. B against each A instance. Report all five A instances, not the best one. The spread across A's seeds is the answer to "how much does the choice of failure matter?"
  3. C against B, overall and by category.
  4. The val-to-test gap for each winner (H2).

Several comparisons on the same data raise the chance of a false positive. Name one primary comparison in PROTOCOL.md before you run anything, for example "B against the mean of the A instances on test". Treat everything else as exploratory.

9.1. Results table template

VariantTest pass rate (task mean)Δ vs control, 95% CIVal scoreVal − test gapRegressionsMedian stepsSignal present
control———————
A (seed 1…5)———————
B———————
C———————

Fill in from your own runs. The dashes are placeholders, not results.

9.2. How to read possible outcomes

10. Verification checklist

Before you trust your own results, check the following:

A quick check for leaks between splits:

jq -r .id split_val.jsonl split_test.jsonl > heldout_ids.txt
grep -F -f heldout_ids.txt evidence_*.json && echo "LEAK" || echo "clean"

11. Failure cases

The proposer rewrites the whole prompt

Without a size limit, the proposer in Arm B may produce a new prompt rather than a change to the old one. You then compare "a new prompt" with "one rule". Enforce the 15-line limit in the linter.

Clusters are an artefact of the labeller

If the labelling model puts 70% of failures in one category, check whether that category is too broad, or whether the labeller defaults to it when it cannot decide. The human check in 6.2 exists for exactly this reason.

The competitor wins with a tool you do not have

If the competitor succeeds because it has a browser or a language server, then a prompt edit cannot copy its success. Mark such trajectories when you build Arm C's input, or exclude them. Otherwise the proposer writes rules that refer to capabilities your agent lacks.

Mutations interact with limits

A rule that adds steps ("first reproduce, then trace, then test") can push the agent past max_steps on large tasks. In the results this shows up as more stop_reason: step_limit. Report the distribution of stop reasons for each variant.

The model behind the API changed mid-experiment

If the control's pass rate on test differs noticeably from the baseline's earlier result on test, something in the environment has changed. Do not compare across that boundary. Re-run all variants in one window.

Success through the test, not the fix

A mutation that says "make sure your test passes" can push the agent to write tests that check its own mistake. If the hidden tests are good, this will not raise the pass rate. If they are weak, it will. Read a sample of successful diffs from each winner. Do not trust the number alone.

Secrets in trajectories

Trajectories contain tool output, and tool output can contain tokens from environment variables, paths with usernames, or customer data from test fixtures. Before you hand trajectories to a proposer or a labeller from an external provider, filter them for secrets. Never publish raw trajectories with an article.

12. Limitations

Summary

A fix built from one failure is a hypothesis drawn from a sample of one. An archive of runs gives you three things a single failure cannot: how often a task fails, a cause shared across tasks, and a held-out check that the fix generalises. A competitor's trajectory adds an example of what success looks like where yours failed. The protocol here lets you check all of this on your own agent: fixed splits, the same proposer for every arm, a size limit on mutations, selection on val, a single final run on test, and a paired comparison over tasks. If the result is "no difference", that is still worth knowing. It tells you to look somewhere other than the prompt.

What to read next

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →