Проверяем Specification Frame на реальной задаче

25 min read · Level: from scratch · Updated October 19, 2026

"Write a function that splits a discount across invoice lines" is one sentence. A large language model will answer it with code that looks correct, passes the example you have in your head, and still loses a cent on one invoice in a few hundred. The sentence never said "integer cents", "the sum must match exactly", or "what happens when the discount exceeds the total". In this article we turn that sentence into a Specification Frame, a prompt with four explicit blocks: context, constraints, output contract and checks. Then we build a small harness that compares it with the short prompt on the same task, using the same tests and an eight-item checklist you can rerun yourself.

1. The problem: short prompts drop requirements silently

A short request carries only the requirements the author wrote down. Everything else is filled in by the model's defaults. For most business code, the defaults are where the bugs live:

• Units and types. If nobody says "cents as integers", float and round() are a perfectly plausible choice.
• Invariants. "Proportionally" does not say that the parts must add up to the whole. Rounding each part independently breaks that.
• Error behaviour. Negative amounts, a discount above the total, empty input: the model picks something, and every run may pick differently.
• Result format. Without a format, you get prose, several alternatives or a code block with extra demo code that runs on import.
• Done criteria. Without stated checks, neither the model nor the reviewer has a shared definition of "correct".

The code that comes back is plausible, and plausible code is exactly what slips through review. The fix is not a longer prompt for its own sake. It is a prompt that states the requirements you would otherwise check in your head, in a form that maps one-to-one onto tests.

2. The concrete case: allocating a discount in cents

The task is small enough to read in one sitting and strict enough to have one correct answer:

An invoice has line amounts in integer cents. A promo discount (also in cents) must be split across the lines proportionally to their amounts, so that refunds and taxes can be computed per line. The per-line discounts are written to an accounting ledger.

The hidden requirements a finance team would insist on:

1. Integer arithmetic only. No floats anywhere.
2. The sum of line discounts equals the invoice discount exactly.
3. No line receives a negative discount or more than its own amount.
4. The leftover cents after flooring are distributed by a deterministic rule. We use the largest remainder method: one cent each to the lines with the largest remainder, with ties going to the lower index.
5. Invalid input raises ValueError, and the input list is never mutated.

The short prompt contains none of this. The Specification Frame contains all of it. The A/B test asks a narrow question: when the requirements are written down in a structured frame, does the generated code satisfy more of them? That is a test of how well a prompt transfers requirements. It is not a test of model intelligence.

3. Anatomy of a Specification Frame

The frame has four blocks. Each block answers one question the model would otherwise have to guess.

A useful rule of thumb: every sentence in the constraints and contract blocks should correspond to at least one test. If a sentence cannot be tested, it is either context or noise.

4. Set up the lab

You need Python 3.11+ and, for automatic generation, an API key for a model provider. The generation script below uses the Anthropic SDK. You can also paste prompts into any chat interface by hand and save the answers as files. The grading step does not care where the code came from.

mkdir spec-frame-lab && cd spec-frame-lab
python3 -m venv .venv
. .venv/bin/activate
pip install pytest anthropic
mkdir prompts candidates raw reports
touch candidates/__init__.py

Final layout:

spec-frame-lab/
├── prompts/
│   ├── short.md        # variant A
│   └── frame.md        # variant B: Specification Frame
├── reference.py        # known-correct implementation
├── test_allocate.py    # the checklist as tests (c1–c8)
├── generate.py         # asks the model N times per variant
├── grade.py            # runs tests per candidate, writes results.csv
├── candidates/         # one .py file per generated answer
├── raw/                # full model responses, for audit
└── reports/            # JUnit XML per candidate

5. Variant A: the short prompt

To keep the comparison fair, the short prompt still names the function and its arguments. Otherwise we would mostly be measuring whether the model guessed the name.

prompts/short.md:

Write a Python function allocate_discount(lines, discount) that distributes
a discount across invoice lines proportionally to their amounts and returns
the per-line discounts as a list.

6. Variant B: the Specification Frame

prompts/frame.md:

## Context
We run a billing service. Invoice line amounts are stored as integer cents
(minor currency units). A promo discount must be split across the lines so
that refunds and taxes can be computed per line. The result goes into an
accounting ledger: if the per-line discounts do not sum to the invoice
discount to the cent, month-end reconciliation is blocked.

## Task
Implement allocate_discount(lines, discount) in Python 3.11, standard
library only.

## Constraints
- lines: list[int], amounts in cents, each >= 0.
- discount: int, amount in cents.
- Integer arithmetic only. Do not use float, Decimal, round() or division
  with "/".
- Do not mutate the input list.
- bool values are invalid (bool is a subclass of int).
- Raise ValueError if: any element of lines or discount is not an int or is
  a bool; discount < 0; any line < 0; discount > sum(lines).
- No I/O, no printing, no code that runs on import.

## Output contract
- Return a list[int] with the same length as lines.
- sum(result) == discount exactly.
- 0 <= result[i] <= lines[i] for every i.
- Base share of line i: (discount * lines[i]) // total, where total = sum(lines).
- Leftover cents (discount - sum of base shares) go one each to the lines
  with the largest remainder (discount * lines[i]) % total; ties go to the
  lower index.
- If discount == 0, return a list of zeros (this also covers lines == []
  and total == 0).
- Response format: exactly one ```python code block containing the whole
  module and nothing else. No text outside the code block.

## Checks
Before answering, verify your code against these cases:
- allocate_discount([100, 200, 300], 100) == [17, 33, 50]
- allocate_discount([1, 1, 1], 2) == [1, 1, 0]
- allocate_discount([10**18, 10**18 + 1], 10**18) sums to exactly 10**18
- allocate_discount([100], 101) raises ValueError
- allocate_discount([True, 2], 1) raises ValueError
- allocate_discount([], 0) == []

Notice what the frame does not contain: no "you are an expert", no motivational filler. Every constraint and contract line maps onto a test in section 8.

7. A reference implementation and a negative control

Before trusting a test suite to grade model output, you need to show that it passes correct code and fails plausible wrong code. This step is easy to skip, and skipping it means you cannot interpret your numbers.

reference.py, the positive control:

def allocate_discount(lines, discount):
    for x in [*lines, discount]:
        if type(x) is not int:
            raise ValueError("amounts must be int cents")
    if discount < 0 or any(x < 0 for x in lines):
        raise ValueError("amounts must be non-negative")
    total = sum(lines)
    if discount > total:
        raise ValueError("discount exceeds invoice total")
    if discount == 0:
        return [0] * len(lines)
    base = [discount * x // total for x in lines]
    rems = [discount * x % total for x in lines]
    leftover = discount - sum(base)
    order = sorted(range(len(lines)), key=lambda i: (-rems[i], i))
    for i in order[:leftover]:
        base[i] += 1
    return base

Why the bounds hold: since discount <= total, each base share is at most lines[i]. A line only gets an extra cent if its remainder is positive, which means its exact share was not an integer. In that case floor + 1 still does not exceed lines[i]. The leftover is always smaller than the number of lines with a positive remainder, so extra cents never go to lines whose share was exact.

Negative control: the kind of code a short prompt can plausibly produce. Save it as candidates/negctl_1.py:

def allocate_discount(lines, discount):
    total = sum(lines)
    return [round(discount * x / total) for x in lines]

This version returns [17, 33, 50] for the obvious example, because 16.67, 33.33 and 50.0 round to the right numbers. That is the trap: it passes the happy path. On [1, 1, 1] with a discount of 2, every line rounds 0.67 up to 1 and the total becomes 3.

Copy the reference in as the positive control:

cp reference.py candidates/posctl_1.py

8. The checklist as tests

Each test name starts with a checklist ID (c1…c8). The grader uses that ID to score items. A parametrized item passes only if all of its cases pass. The test file contains its own test oracle: an independent copy of the allocation rule used for randomized comparison.

test_allocate.py:

import importlib
import os
import random

import pytest

mod = importlib.import_module(os.environ.get("CANDIDATE", "reference"))
allocate = mod.allocate_discount


def oracle(lines, discount):
    # Same rules as the output contract, kept independent of the candidate.
    total = sum(lines)
    if discount == 0:
        return [0] * len(lines)
    base = [discount * x // total for x in lines]
    rems = [discount * x % total for x in lines]
    order = sorted(range(len(lines)), key=lambda i: (-rems[i], i))
    for i in order[: discount - sum(base)]:
        base[i] += 1
    return base


def random_case(rng):
    lines = [rng.randint(0, 10_000) for _ in range(rng.randint(1, 8))]
    return lines, rng.randint(0, sum(lines))


def test_c1_basic_example():
    assert allocate([100, 200, 300], 100) == [17, 33, 50]


def test_c2_sum_is_exact_on_large_values():
    lines = [10**18, 10**18 + 1, 3]
    result = allocate(lines, 10**18)
    assert sum(result) == 10**18
    assert result == oracle(lines, 10**18)


def test_c3_returns_ints_only():
    result = allocate([333, 333, 334], 100)
    assert all(type(x) is int for x in result)


def test_c4_length_and_bounds():
    rng = random.Random(7)
    for _ in range(300):
        lines, discount = random_case(rng)
        result = allocate(lines, discount)
        assert len(result) == len(lines)
        assert sum(result) == discount, (lines, discount)
        assert all(0 <= a <= b for a, b in zip(result, lines)), (lines, discount)


def test_c5_remainder_rule_and_ties():
    assert allocate([1, 1, 1], 2) == [1, 1, 0]
    rng = random.Random(42)
    for _ in range(500):
        lines, discount = random_case(rng)
        assert allocate(lines, discount) == oracle(lines, discount), (lines, discount)


@pytest.mark.parametrize(
    "lines, discount",
    [
        ([100], 101),
        ([100], -1),
        ([-5, 10], 1),
        ([True, 2], 1),
        ([1.5, 2], 1),
        ([100], 1.0),
        ([], 1),
    ],
)
def test_c6_rejects_invalid_input(lines, discount):
    with pytest.raises(ValueError):
        allocate(lines, discount)


def test_c7_edge_cases():
    assert allocate([], 0) == []
    assert allocate([0, 0], 0) == [0, 0]
    assert allocate([0, 500], 100) == [0, 100]


def test_c8_does_not_mutate_input():
    lines = [100, 200, 300]
    snapshot = list(lines)
    allocate(lines, 100)
    assert lines == snapshot

The checklist in plain words, so it can also be used in manual code review:

Validate the tests before using them

CANDIDATE=reference python -m pytest -q test_allocate.py
CANDIDATE=candidates.negctl_1 python -m pytest -q --tb=line test_allocate.py

Expected behaviour, which you should confirm on your machine: the reference passes everything. The negative control passes c1 and fails at least c3, c5, c6 and c7. It fails c3 because round() of a float division returns an int in Python 3, but [True, 2]-style inputs are not rejected, and [0, 0] divides by zero. Trace each failure to a line of the negative control before you go on. If the reference fails anything, fix the tests first. Never "fix" the reference to match the tests.

9. Generate candidates

generate.py asks the model RUNS times per variant and saves both the extracted code and the raw response. The API key is read from the environment by the SDK. Never put it in the script or the article.

import os
import pathlib
import re

import anthropic

MODEL = os.environ.get("MODEL", "claude-sonnet-5-5")
RUNS = int(os.environ.get("RUNS", "10"))

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
code_dir = pathlib.Path("candidates")
raw_dir = pathlib.Path("raw")

for variant in ("short", "frame"):
    prompt = pathlib.Path(f"prompts/{variant}.md").read_text()
    for i in range(1, RUNS + 1):
        msg = client.messages.create(
            model=MODEL,
            max_tokens=4000,
            messages=[{"role": "user", "content": prompt}],
        )
        text = "".join(b.text for b in msg.content if b.type == "text")
        (raw_dir / f"{variant}_{i}.md").write_text(text)
        match = re.search(r"```(?:python)?\n(.*?)```", text, re.S)
        code = match.group(1) if match else text
        (code_dir / f"{variant}_{i}.py").write_text(code)
        print(f"{variant}_{i}: {'code block' if match else 'NO code block'}")

export ANTHROPIC_API_KEY=...   # set in your shell, not in files
MODEL=claude-sonnet-5-5 RUNS=10 python generate.py

Both variants use the same model, the same max_tokens and default sampling. Write down the model ID and the date of the run. A later rerun against an updated model is a different experiment.

10. Grade: run the checklist for every candidate

Generated code is untrusted code. The grader runs it, so run grading in a throwaway container or VM, with no network and no secrets in the environment. The script also strips the API key from the subprocess environment, so that a candidate cannot read it.

grade.py:

import csv
import os
import pathlib
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
from collections import defaultdict

ITEMS = [f"c{i}" for i in range(1, 9)]
rows = []

for path in sorted(pathlib.Path("candidates").glob("*_*.py")):
    if path.name.startswith("__"):
        continue
    variant = path.stem.rsplit("_", 1)[0]
    report = pathlib.Path("reports") / f"{path.stem}.xml"
    report.unlink(missing_ok=True)
    env = {**os.environ, "CANDIDATE": f"candidates.{path.stem}"}
    env.pop("ANTHROPIC_API_KEY", None)
    try:
        subprocess.run(
            [sys.executable, "-m", "pytest", "-q", "--tb=no", "-p", "no:cacheprovider",
             f"--junitxml={report}", "test_allocate.py"],
            env=env, capture_output=True, timeout=120,
        )
    except subprocess.TimeoutExpired:
        pass

    passed = {item: True for item in ITEMS}
    seen = set()
    if report.exists():
        for case in ET.parse(report).iter("testcase"):
            m = re.match(r"test_(c\d+)_", case.get("name", ""))
            if not m:
                continue
            seen.add(m.group(1))
            if any(case.find(tag) is not None for tag in ("failure", "error", "skipped")):
                passed[m.group(1)] = False
    for item in ITEMS:
        if item not in seen:  # import error, timeout, missing function
            passed[item] = False

    rows.append({"candidate": path.stem, "variant": variant,
                 **{k: int(v) for k, v in passed.items()},
                 "score": sum(passed.values())})

with open("results.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["candidate", "variant", *ITEMS, "score"])
    writer.writeheader()
    writer.writerows(rows)

by_variant = defaultdict(list)
for row in rows:
    by_variant[row["variant"]].append(row)
for variant, group in sorted(by_variant.items()):
    n = len(group)
    items = "  ".join(f"{k}:{sum(r[k] for r in group)}/{n}" for k in ITEMS)
    full = sum(r["score"] == len(ITEMS) for r in group)
    print(f"{variant:8} n={n:2}  {items}  all-pass:{full}/{n}")

unset ANTHROPIC_API_KEY
python grade.py

Output has one line per variant: negctl, posctl, short, frame. Check the controls first. posctl must be all-pass and negctl must not be. If either is off, your numbers for short and frame mean nothing.

11. Record and read the results

Fill this table from your own results.csv. We leave it empty on purpose.

Also record: model ID, date, RUNS, SDK version (pip show anthropic), and the git hash or checksum of both prompt files.

How to read it without overclaiming:

• Report counts, not percentages, for small N. "7/10 vs 10/10" is honest. "70% vs 100%" invites more confidence than ten samples support.
• Look per item, not only at the total. The useful finding is which requirement the short prompt drops, for example the tie-break rule or input validation, because that tells you what to put in future frames.
• Read the failing code. Open raw/ for every failed item. A TypeError instead of ValueError is a different problem from a wrong sum.
• Expect the short prompt to fail some items "by design". It was never told about ValueError or the tie-break. A failure there shows the requirement was not transferred, and that is the effect we are measuring. It is not evidence that the model is weak.

12. Score the prompt itself

The tests grade the code. The following checklist grades the prompt before you spend API calls on it. Score each line 0 or 1. Two reviewers should arrive at the same score, and if they disagree, the line is ambiguous and needs rewording.

1. Context states who consumes the result and what a failure costs.
2. Language, version and allowed dependencies are named.
3. Units and types of every input are explicit.
4. Forbidden constructs are listed (here: float, Decimal, round(), /).
5. Every error condition names the exception type.
6. The return type and length are stated.
7. Every invariant is written as a checkable expression (for example sum(result) == discount).
8. Any ambiguous algorithm choice (rounding, ties, ordering) is resolved.
9. The response format is specified precisely enough to parse automatically.
10. At least one check covers an edge case, not only the happy path.
11. Each constraint/contract sentence maps to at least one test ID.
12. The prompt contains no secrets, customer data or real account identifiers.

The short prompt in section 5 scores at most 1/12 (item 12). The frame in section 6 is designed to score 12/12. Re-score it yourself instead of taking that on trust.

13. Failure cases and how to handle them

• No code block in the answer. generate.py falls back to the full text, the import fails and every item scores 0. Count these separately as format failures. Mixing them into "logic failures" hides a different problem.
• Demo code on import. A candidate with print(allocate_discount(...)) at module level still imports. Code that reads input() blocks until timeout. The 120-second timeout plus a closed stdin (from capture_output) keeps the grader moving.
• Wrong exception type. TypeError for [1.5, 2] fails c6. That is correct per the contract. If you think TypeError should be acceptable, change the contract and the test together, then rerun both variants.
• Different but valid tie-break. A candidate that gives leftover cents to the largest lines instead of the largest remainders can still satisfy c4 and fail c5. The contract chose one rule because ledgers need reproducibility. A different rule is a requirement miss, not a bug in the test.
• Rate limits or API errors. If generation stops halfway, the variants end up with unequal N. Rerun the missing indices instead of comparing 10 against 6.
• Executing untrusted code. The grader runs whatever the model wrote. Run it in an isolated environment with no network and no credentials. env.pop removes one key, but it is not a sandbox.
• Prompt injection via your own inputs. If you later build frames from tickets or user text, that text can carry instructions. Keep untrusted material inside a clearly delimited context block and never let it override constraints. See the prompt injection guide.

14. Limitations of this experiment

• One task, one domain. Money allocation has a single correct answer, which makes it easy to grade. Tasks with several acceptable designs need rubric-based review on top of tests.
• Overlap between prompt and tests. The frame's checks literally contain the c1 and c5 tie examples. Some improvement may come from the model copying examples rather than understanding rules. To separate the two, add a third variant frame_noex with the Checks block removed, and compare.
• Same author for frame and tests. Whoever writes both can unconsciously "teach to the test". Ask someone else to write extra hidden tests from the context paragraph alone.
• Sampling variance and model drift. Small N and default sampling mean results shift between runs. A model update can change everything. Treat results as dated observations.
• Prompt length is a confound. The frame is both longer and more structured. This setup cannot tell you whether the structure or simply the extra information is responsible. A fair follow-up is the same requirements written as an unstructured paragraph.

15. Reusing the method on your own task

1. Write the one-sentence request you would normally send.
2. List the requirements a reviewer would check. Each becomes a checklist ID.
3. Write the tests and a reference implementation. Validate them with a positive and a negative control.
4. Write the frame: context, constraints, output contract, checks. Map each sentence to a test ID.
5. Score the prompt with the 12-item checklist.
6. Generate N candidates per variant with the same model and settings, then grade in isolation.
7. Read failures per item, update the frame, and rerun. Keep old prompt versions in git so each result can be traced to an exact prompt.

The point is not that structured prompts are magic. Requirements that are not written down cannot be met reliably, and requirements that are written down but never tested cannot be shown to be met. A Specification Frame paired with a checklist covers both.

Further reading: browse the practical guides, including testing AI agents. Look up any term in the glossary.