Skills for a Coding Agent: Measuring Their Value on a Real Task

What you will produce

You will compare two modes of the same coding agent while keeping the model, repository, task, and available tools unchanged:

1. Mode A — general-purpose. The agent receives the task and the repository’s normal instructions, but not the experimental skills.
2. Mode B — skill-equipped. The agent receives the identical task and can apply three local guides: evidence-based review, behavior-oriented test design, and contract documentation synchronization.

The experiment should leave these artifacts:

• three installed skill files;
• an immutable copy of the task prompt;
• separate working directories for modes A and B;
• agent responses, patches, command output, and run logs;
• a manual review-completeness score;
• test execution and negative-control results;
• a documentation score;
• measured wall-clock time;
• input and output token usage when the runtime exposes it;
• a comparison table with no predetermined winner.

What skills are meant to fix

A universal agent can read code, execute commands, edit files, and explain a patch. However, “do a thorough job” does not define which checks are mandatory. One run may begin with tests, another with implementation, and a third may stop after describing a possible defect.

In this laboratory, a skill is a small versioned document containing activation conditions, a sequence of actions, completion criteria, and prohibitions. It is neither a separate model nor a hidden knowledge base. It organizes capabilities that the agent already has.

A skill is also different from a long prompt. The prompt states the current objective; the skill describes a reusable method. If a method applies across many tasks, store and version it separately, then activate it only when its conditions match.

The concrete case: reserving inventory

A useful comparison task must involve code, tests, and public documentation. Use a small training service or isolate a similarly sized module in your own repository. The example function reserves every line in an order:

def reserve_order(inventory, order):
    reserved = []

    for item in order.items:
        inventory.reserve(item.sku, item.quantity)
        reserved.append(item.sku)

    return {
        "order_id": order.id,
        "reserved": reserved,
    }

Give both agent modes this exact task:

Review the inventory reservation function, fix the problems you confirm, add the necessary tests, and update the documentation of its public behavior. Do not change the successful response format unless necessary. Run appropriate checks and report what was verified.

Before either run, a human evaluator records the known requirements:

1. Quantity must be a positive integer.
2. The same SKU may occur more than once in one order.
3. If any reservation fails, earlier reservations must be compensated.
4. An error must not produce a partially successful response.
5. An empty order must follow an explicitly selected contract.
6. Documentation must explain operation atomicity and expected errors.

These requirements form the evaluator’s reference sheet. Do not give the sheet to the agent. Otherwise, the experiment measures the ability to reproduce a supplied checklist instead of the value of the skills. For your own task, write an equivalent reference sheet before the first run.

Why this case works

The change is small enough to run twice, but it requires several kinds of reasoning. A superficial response may add only a successful-order test. A more complete review may identify the partial side effect, repeated items, quantity validation, and the need to document the failure contract.

Use local test fixtures and a fake inventory adapter. Do not include production credentials, real customer data, or uncontrolled network dependencies.

Experiment design

Treat the comparison as a small benchmark. The availability of the skills should be the only intentional difference between modes.

Freeze the controlled conditions

• the same model and model version;
• the same generation parameters, when configurable;
• the same starting commit;
• the same task text, byte for byte;
• the same command and tool-calling permissions;
• the same network policy;
• the same time and step limits;
• equally cold or equally warmed dependency caches;
• separate worktrees with no artifacts from the other mode.

Create independent worktrees

Do not run the two modes sequentially in the same working directory. The second run must not see the first run’s code, tests, notes, or logs.

mkdir -p experiment
git rev-parse HEAD > experiment/base-sha.txt

git worktree add ../skills-exp-baseline "$(cat experiment/base-sha.txt)"
git worktree add ../skills-exp-skilled  "$(cat experiment/base-sha.txt)"

If project-level skills must reside inside the repository, either store them outside both worktrees or create a common commit containing them and explicitly disable their loading in mode A. Record the exact disabling mechanism in the report.

Control the run order

For one pair, run A then B and record that order. For a larger set, alternate it: A–B for the first task, B–A for the second. This reduces bias from warm caches, provider load, and the evaluator remembering the preceding response.

Create the quality rubric first

Define all scoring rules before opening either result. Otherwise, a polished answer can unconsciously change what the evaluator considers important.

Review completeness

Create experiment/review-rubric.csv:

id,requirement,weight,evidence_rule
R1,"Compensate earlier reservations after a later failure",3,"Names the partial-reservation scenario and points to the responsible code"
R2,"Validate a positive integer quantity",2,"Identifies at least one concrete invalid input"
R3,"Define behavior for a repeated SKU",2,"Considers aggregation or sequential handling"
R4,"Define the empty-order contract",1,"States and verifies the selected behavior"
R5,"Prevent a partially successful response",2,"Connects the exception path to the public contract"
R6,"Synchronize public documentation",1,"Covers atomicity and documented errors"

Weight 3 represents a data-integrity risk, weight 2 represents significant behavior, and weight 1 represents contract completeness. Do not adjust weights after seeing the answers.

Test quality

Documentation quality

Award one point for each accurately documented item: public entry point, valid input, successful result, partial-failure behavior, empty-order behavior, and the test command. The maximum documentation score is 6.

The three-skill set

Skill locations and manifests vary by agent. The portable structure below uses one directory per skill and a SKILL.md file containing metadata and instructions. If your environment uses another manifest, preserve the activation conditions and procedure.

skills-src/
├── evidence-code-review/
│   └── SKILL.md
├── behavior-test-design/
│   └── SKILL.md
└── contract-doc-sync/
    └── SKILL.md

Skill 1: evidence-based code review

Create skills-src/evidence-code-review/SKILL.md:

---
name: evidence-code-review
description: Apply when reviewing a code change or investigating defects before editing.
---

# Evidence-based code review

1. Read the task, the diff, and applicable repository instructions first.
2. List changed public contracts, side effects, and data boundaries.
3. For every risk, state a failure scenario:
   input → state → action → observable incorrect result.
4. Verify each hypothesis against code, tests, and configuration.
5. Do not report a finding without a concrete location and evidence.
6. Separate:
   - confirmed defects;
   - risks requiring more information;
   - style improvements.
7. Before editing, write a short repair plan.
8. After editing, recheck every confirmed scenario.

Required risk classes:
- validation errors and boundary values;
- partial side effects and compensation;
- repeated and concurrent operations;
- exception handling;
- changes to external contracts;
- security and trust boundaries.

Finding format:
- severity;
- file and line;
- failure scenario;
- evidence;
- minimal verification or repair.

Do not expand the review into unrelated old code unless it is
needed to verify changed behavior.

Skill 2: behavior-oriented test design

Create skills-src/behavior-test-design/SKILL.md:

---
name: behavior-test-design
description: Apply when adding, repairing, or evaluating automated tests.
---

# Behavior-oriented test design

1. Before writing tests, create a matrix:
   behavior, input, precondition, observation, expected result.
2. Cover at least:
   - a representative successful path;
   - input boundaries;
   - failure of an external dependency;
   - a partially completed sequence;
   - empty or missing input when applicable;
   - a repeated operation when side effects exist.
3. Assert public behavior and meaningful side effects.
4. Avoid coupling tests to internal implementation without a reason.
5. Do not weaken existing expectations to obtain a green run.
6. Run each new defect test against the original implementation first.
   It must fail for the expected reason.
7. Apply the smallest repair, then run:
   - the new focused test;
   - the module test suite;
   - the complete available suite.
8. If the complete suite cannot run, report the exact reason.

Final report:
- scenarios added;
- commands executed;
- observed pass and failure counts;
- anything left unverified.

Skill 3: contract documentation synchronization

Create skills-src/contract-doc-sync/SKILL.md:

---
name: contract-doc-sync
description: Apply when code changes public behavior, errors, configuration, or usage commands.
---

# Contract documentation sync

1. Identify the audience and public entry point.
2. Compare the old and new contracts:
   inputs, outputs, errors, side effects, and limitations.
3. Find documentation, examples, and comments describing the contract.
4. Document only behavior confirmed by code and tests.
5. Do not promise guarantees that the implementation does not provide.
6. For each usage example, verify:
   - function and parameter names;
   - result format;
   - expected error;
   - execution command.
7. Run executable examples or at least validate their syntax.
8. List updated documents and known limitations in the final report.

Documentation is complete when a reader can determine:
- what the operation does;
- which inputs are valid;
- what success means;
- what happens on failure;
- how to verify the described behavior.

The files are intentionally short. A skill that controls architecture, security, Git, testing, and documentation at once becomes another vague global prompt.

Installation and discovery

Find the documented user-level or project-level skill directory for your agent. Do not guess: some agents read skills directly from a repository, others use a user directory, and some require registration in configuration.

If your agent supports .agent/skills, installation can look like this:

install -d .agent/skills

cp -R skills-src/evidence-code-review .agent/skills/
cp -R skills-src/behavior-test-design .agent/skills/
cp -R skills-src/contract-doc-sync .agent/skills/

find .agent/skills -maxdepth 2 -name SKILL.md -print

The command should print exactly three paths:

.agent/skills/evidence-code-review/SKILL.md
.agent/skills/behavior-test-design/SKILL.md
.agent/skills/contract-doc-sync/SKILL.md

Next, use the agent’s supported skill-listing command or interface. Record the actual command and its output. Do not substitute an example command from a different product.

Discovery checklist:
[ ] evidence-code-review is listed
[ ] behavior-test-design is listed
[ ] contract-doc-sync is listed
[ ] each description is visible to the agent
[ ] all three are disabled in mode A
[ ] all three are available in mode B

Verify activation, not only file presence

A correctly placed file may still be ignored because of invalid metadata. Give the agent a read-only diagnostic request: “Which available skills apply to a task involving review, tests, and documentation? Name them without changing files.”

Mode B should identify all three installed names. Mode A must not expose them. Save both diagnostic responses. If the modes see the same skills, stop: the intended treatment was not applied.

Mode A: run the general-purpose agent

Treat the unskilled run as the baseline. Enter its worktree, verify the starting commit, and confirm that the existing suite is green:

cd ../skills-exp-baseline
git status --short
git rev-parse HEAD
python -m pytest

If the starting suite already fails, record the known failures or select another commit. Do not attribute a pre-existing failure to either agent mode.

Store the task in one immutable file:

sha256sum experiment/task.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/baseline-started-at.txt

Send the exact contents of experiment/task.txt without extra hints. Save the complete response, patch, command log, model identifier, runtime limits, and usage information exposed by the environment.

git diff --binary > experiment/baseline.patch
git diff --stat > experiment/baseline-stat.txt
git status --short > experiment/baseline-status.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/baseline-finished-at.txt

Mode B: run the skill-equipped agent

Repeat the same preparation in the skilled worktree. Confirm that its initial SHA matches the recorded base and that the three skills are discoverable.

cd ../skills-exp-skilled
git status --short
test "$(git rev-parse HEAD)" = "$(cat ../agentlabjournal/experiment/base-sha.txt)"
python -m pytest

Send the same task file without adding “use every skill” or naming expected defects. Skill activation is part of the behavior being tested.

sha256sum experiment/task.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/skilled-started-at.txt

# Run the agent through the normal interface.

git diff --binary > experiment/skilled.patch
git diff --stat > experiment/skilled-stat.txt
git status --short > experiment/skilled-status.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" > experiment/skilled-finished-at.txt

Preserve any trace showing which skills were loaded. A final response claiming that a skill was used is weaker evidence than a runtime event or discovery record.

Measure time and context usage

Prefer timestamps emitted by the agent runtime. If none are available, wrap the complete invocation with a monotonic timer or GNU time:

/usr/bin/time -f \
'elapsed_seconds=%e
user_seconds=%U
system_seconds=%S
max_rss_kb=%M' \
-o experiment/timing.txt \
your-agent-command < experiment/task.txt

Measure the entire agent run, but report dependency installation separately if only one mode had to perform it.

What “context usage” means

Record distinct values rather than merging them:

• uncached input tokens;
• cached input tokens, if reported;
• output tokens;
• total model calls;
• peak context size, if available;
• number and type of tool calls;
• bytes or lines read from repository files as a secondary proxy.

{
  "mode": "A-or-B",
  "model": "copy-from-runtime",
  "started_at": "copy-from-log",
  "finished_at": "copy-from-log",
  "elapsed_seconds": null,
  "input_tokens_uncached": null,
  "input_tokens_cached": null,
  "output_tokens": null,
  "peak_context_tokens": null,
  "model_calls": null,
  "tool_calls": null,
  "usage_source": "runtime-log-or-unavailable"
}

Leave unavailable values as null or label them “not available.” Do not estimate token counts from character length and present them as provider-reported usage.

Score review completeness

Blind the variants before evaluation. Copy the responses and patches to neutral names such as variant-x and variant-y, then score each rubric row.

id,weight,variant_x_found,variant_y_found,x_evidence,y_evidence
R1,3,0,0,"",""
R2,2,0,0,"",""
R3,2,0,0,"",""
R4,1,0,0,"",""
R5,2,0,0,"",""
R6,1,0,0,"",""

Calculate weighted completeness:

review_completeness =
  sum(weight for confirmed applicable findings)
  / sum(weight for all applicable rubric findings)

A finding counts only if it describes an applicable failure scenario and cites enough code evidence to verify it. Vague advice such as “improve error handling” does not satisfy R1 or R5.

Track false findings separately

A longer review can appear more complete simply because it makes more claims. For every reported issue, classify it as confirmed, unconfirmed because information is missing, or contradicted by the repository.

finding_precision =
  confirmed_findings / all_checkable_findings

Do not force an opinion-based suggestion into the defect category. Style recommendations may be useful, but they do not prove review accuracy.

Score test quality

A green test file is not enough. Evaluate executability, defect detection, observed behavior, and stability.

1. Executability

python -m pytest -q path/to/new_test_file.py
python -m pytest -q path/to/module_tests
python -m pytest -q

Save raw output and exit status for each command.

2. Negative control

A useful regression test should expose the behavior it claims to protect. Apply the new tests to a clean copy of the original implementation:

git worktree add ../skills-exp-control "$(cat experiment/base-sha.txt)"
cd ../skills-exp-control

# Copy only the new or modified tests from the evaluated variant.
python -m pytest -q path/to/new_test_file.py

At least one defect-specific test should fail for the expected reason. If every new test passes on the original implementation, the tests may not exercise the repaired behavior.

3. Observable effects

For the second-item failure, verify the compensation call rather than only checking that an exception was raised. For a repeated SKU, assert the selected quantity behavior rather than merely checking that the function returned.

4. Stability

for run in 1 2 3 4 5; do
  python -m pytest -q path/to/new_test_file.py || exit 1
done

Five local repetitions do not prove the absence of flakiness, but they can reveal state leakage, order dependence, random inputs, and uncontrolled time.

Score documentation quality

Compare documentation with the final code and tests. Natural-sounding prose is not evidence that the contract is correct.

If documentation says “the operation is always atomic” while compensation can itself fail, the statement is too strong. Document the actual guarantee and any known limit of the compensation mechanism.

Build the comparative report

Do not complete the table from memory. Every value should point to a response, patch, command output, trace, or scoring sheet.

Write a defensible conclusion

Use four parts:

1. Observation: state which metrics actually differed.
2. Evidence: identify the saved artifacts supporting the values.
3. Interpretation: explain which skill may have contributed to the difference.
4. Decision: state where the set should be used and where its cost is not justified.

On this task, mode B confirmed [number] of [number] applicable risks, while mode A confirmed [number]. The additional findings were verified by tests [identifiers]. Mode B used [measured value] uncached input tokens and [measured value] seconds, compared with [values] for mode A. We therefore accept the skill set for [tested task class], but not yet for [untested task class].

If no meaningful difference appears, report that result. Possible explanations include an already detailed project prompt, a task that was too simple, failed skill activation, or a model that already followed the same method consistently.

Verify the experiment

The saved artifacts provide basic observability: they let another reader reconstruct what the agent saw, changed, and executed. Before accepting the conclusion, verify:

1. Both worktrees started from the same SHA.
2. The task files have identical hashes.
3. The model and configurable parameters were unchanged.
4. Mode A could not read the experimental skills.
5. Mode B discovered the intended skills.
6. The reference requirements were written before either run.
7. The rubric and weights were not changed after reading the outputs.
8. Every reported number comes from a saved record.
9. New tests were exercised against the original implementation.
10. The full available suite ran for both variants.
11. Documentation was compared with behavior, not only proofread.
12. Unavailable measurements are explicitly marked unavailable.

Repeatability check

One paired run demonstrates one case but says little about consistency. If the budget allows, repeat both modes three times from clean states. Do not select the best response from each set. Report median time and token usage, the quality range, and the fraction of runs that confirmed each critical risk.

finding_consistency =
  runs_where_finding_was_confirmed / total_runs

This consistency value directly addresses the original problem: the agent’s review, testing, and documentation behavior changes between runs.

Failure cases

The files are installed, but the skills never activate

Common causes are an incorrect directory, invalid metadata, or an activation description that is too narrow. Test discovery and a read-only diagnostic request before running the experiment.

The agent loads every skill for every task

This consumes context and dilutes priorities. Narrow the descriptions: the testing skill applies when tests are created or assessed, while the documentation skill applies when a public contract changes.

A skill leaks the answer to the case

A skill that explicitly says “check repeated SKUs and compensate inventory reservations” contains the evaluator’s reference answer. Skills should describe general risk classes such as boundaries, repetitions, partial side effects, and dependency failures.

Mode B gets a more detailed task

One extra instruction such as “make sure the operation is atomic” can explain the entire difference. Store the task once and supply it without manual edits.

Quality is measured by lines changed

A large patch is not necessarily a good repair. The agent may rewrite the module, add brittle mocks, and introduce new risks. Score observable behavior and patch minimality separately.

Green tests are accepted without a negative control

A new test may already pass before the repair. Running it against the original implementation distinguishes a useful defect test from a decorative one.

The final response is treated as execution evidence

“All tests pass” does not replace saved command output. A run may have reached a limit, hidden part of the output, or executed only one test file.

Timing is compared across different environments

The first run may install dependencies or warm caches. Prepare both environments in advance or account for setup time in exactly the same way.

The agent edits the evaluation rubric

Keep experiment definitions outside the writable task worktree or make them read-only. Otherwise, the agent may accidentally alter the criteria used to judge its own result.

Limitations

A single case does not prove universal value. The skills may work well on a small Python service and add friction in a monorepo, mobile application, generated-code project, or infrastructure migration.

Manual review scoring contains judgment. Reduce subjectivity by defining the rubric in advance, hiding variant identities, and involving a second evaluator for disputed findings.

Token consumption depends on more than skill text. Agent implementation, tool schemas, repeated file reads, caching, history compression, and hidden system instructions all affect usage. The comparison is valid only within the frozen environment.

A compensation test against a fake adapter does not prove transactional behavior in a distributed inventory system. A real integration may require idempotent commands, durable queues, reconciliation, or a separate compensation process.

Skills become outdated as the repository changes. A new test framework, documentation format, architecture, or review policy requires a skill update and another comparison.

Agent runs are nondeterministic. Even repeated pairs cannot guarantee future behavior; they provide an empirical range for a defined task class and environment.

Adopting the skills in a real project

If the comparison supports adoption, turn the set into a versioned workflow:

1. Store skill sources beside the code or in a controlled shared repository.
2. Assign an owner to every skill.
3. Record the skill-set version or SHA in run logs.
4. Activate only skills relevant to the current task.
5. Validate skill metadata and discovery in continuous integration.
6. Maintain a small evaluation set of anonymized real tasks.
7. Repeat the controlled comparison after changing a skill.
8. Reject updates that improve completeness by generating too many false findings.
9. Monitor time and uncached input-token usage.
10. Remove instructions that do not change observable behavior.

Practical acceptance rule

Accept the set for the tested task class only when it repeatedly improves at least one important quality metric, does not reduce critical completeness, remains within the chosen false-finding threshold, and fits the project’s time and context budgets.

If the full set improves complex repairs but makes trivial edits unnecessarily expensive, route by risk. Small local changes may not need all three skills. Changes involving side effects, public interfaces, or error handling should activate the relevant methods.

Replication checklist

1. Select a task involving code, tests, and documentation.
2. Record known risks before running the agent.
3. Freeze the model, parameters, SHA, limits, and environment.
4. Create independent worktrees for modes A and B.
5. Install the three skills and verify discovery.
6. Confirm that mode A cannot access them.
7. Send both modes the identical task.
8. Save responses, patches, commands, usage, and raw logs.
9. Run new tests against the original implementation.
10. Score review, tests, and documentation using the predefined rubric.
11. Use only recorded values for time and context usage.
12. Repeat the paired runs to estimate consistency.
13. Accept, restrict, revise, or reject the set based on measured evidence.