Evaluation · Intermediate · 12 min read

How to Measure AI Agent Reliability: Use pass^k Instead of a Single Attempt

Как измерять надежность AI-агента: pass^k вместо одной попытки
Temporary fallback cover; replace in editorial pass.

A one-shot success rate answers whether an agent can complete a task once. It does not answer whether the agent will complete the same task reliably tomorrow, on the next run, or across a batch of similar requests. For recurring actions, measure repeatability with passk: the share of tasks that pass in every one of k independent attempts.

12 minutes Intermediate Updated 16 November 2026

The problem with a single attempt

Suppose an agent succeeds on 90 out of 100 test tasks. A report might say that the agent has a 90% success rate. That number is useful, but it silently assumes that one successful attempt is enough evidence of reliability.

Recurring actions have a stricter requirement. If an agent must execute the same workflow five times, a task is dependable only when all five executions succeed. A run that succeeds once and fails on the fourth retry is not reliable enough for an unattended process.

The gap appears when success is probabilistic. If each attempt has an independent probability p of passing, then the probability that all k attempts pass is:

passk = pk

With a per-attempt success probability of 0.90, five consecutive successful attempts have probability 0.905 = 0.59049 under the independence assumption. The exact value for a real system must be measured from repeated task executions, but the example shows why a one-shot rate can overestimate recurring reliability.

What passk measures

For a fixed evaluation task, run the agent k times. Mark each attempt as either pass or fail using the same deterministic evaluator. The task contributes to passk only if every one of its attempts passes.

Across m tasks, let Ii be 1 when task i passes all k attempts, and 0 otherwise. Then:

passk = (number of tasks passing all k attempts) / m

This is a task-level metric. It does not count every successful attempt separately. For example, if a task passes four times and fails once, it contributes zero to pass5.

passk versus pass@k

pass@k answers a different question: does at least one of k attempts pass? It is appropriate when the system can generate multiple candidates and select a successful one, such as code synthesis with a test-based filter.

pass@k = 1 − P(all k attempts fail)

If there are n sampled attempts and c of them pass, the commonly used unbiased estimator is:

pass@k = 1 − C(n − c, k) / C(n, k)

The distinction is operational:

Metric Question Typical use
passk Do all k executions pass? Reliability of recurring actions
pass@k Does at least one of k attempts pass? Candidate generation and selection

A high pass@k can coexist with a low passk. That is not a contradiction: one metric rewards finding a successful candidate, while the other measures consistency across all required executions.

A reproducible evaluation design

Before writing code, define the unit of evaluation. A task should contain the input, the allowed tools or environment, and a pass/fail evaluator. Keep the evaluator separate from the agent so that the agent cannot decide whether it succeeded.

  1. Choose a fixed task set with m cases.
  2. Choose a repetition count k, such as 3, 5, or 10.
  3. Run every task exactly k times.
  4. Record the attempt output and the evaluator result.
  5. Compute pass1, passk, and optionally pass@k.
  6. Store the model version, prompt version, tool configuration, seed policy, and evaluation date.

If the action changes external state, use a sandbox, transaction rollback, fixture reset, or a mocked tool boundary. Do not allow earlier attempts to change the inputs seen by later attempts unless statefulness is the behavior being measured.

Minimal result format

{
  "task_id": "invoice-017",
  "attempt": 3,
  "passed": true,
  "output": "...",
  "evaluator_version": "invoice-check-v2"
}

Keep the output field optional or redact it when it may contain personal or confidential data. The metric calculation needs the boolean result, not hidden customer information.

Python implementation

The following code calculates passk from a matrix in which each row is one task and each column is one attempt. It uses only the standard library.

from math import comb
from typing import Iterable, Sequence


def pass_k(results: Sequence[Sequence[bool]], k: int) -> float:
    """Fraction of tasks that pass all k attempts."""
    if k < 1:
        raise ValueError("k must be at least 1")
    if not results:
        raise ValueError("results must contain at least one task")

    valid = [row for row in results if len(row) >= k]
    if len(valid) != len(results):
        raise ValueError("every task must have at least k attempts")

    passing_tasks = sum(all(row[:k]) for row in valid)
    return passing_tasks / len(valid)


def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k estimator from n samples and c passing samples."""
    if not (1 <= k <= n):
        raise ValueError("k must be between 1 and n")
    if not (0 <= c <= n):
        raise ValueError("c must be between 0 and n")

    failures = n - c
    if failures < k:
        return 1.0

    return 1.0 - comb(failures, k) / comb(n, k)


results = [
    [True, True, True, True, True],
    [True, True, False, True, True],
    [True, True, True, True, True],
    [False, True, True, True, True],
]

print("pass^1:", pass_k(results, 1))
print("pass^5:", pass_k(results, 5))
print("pass@5:", pass_at_k(n=5, c=4, k=5))

In this illustrative dataset, the first attempt passes for two of four tasks, so pass1 is 0.5. No task passes all five attempts, so pass5 is 0.0. These are example calculations from the values in the code, not measurements of a production agent.

Calculating a reliability curve

A single value of k can hide where reliability declines. Calculate a curve for several repetition counts:

for k in range(1, 6):
    print(f"pass^{k} = {pass_k(results, k):.3f}")

The curve is non-increasing when it is computed from the same ordered attempt matrix: adding a required attempt cannot turn a failed all-attempts condition into a pass. A sharp drop between two values of k is a useful signal for investigating intermittent failures.

Verification checks

A reproducible evaluation should include checks that catch invalid datasets and misleading comparisons.

def verify_results(results: Sequence[Sequence[bool]], max_k: int) -> None:
    previous = 1.0

    for k in range(1, max_k + 1):
        current = pass_k(results, k)

        if not 0.0 <= current <= 1.0:
            raise AssertionError("metric must be between 0 and 1")

        if current > previous:
            raise AssertionError("pass^k must not increase as k increases")

        previous = current


verify_results(results, max_k=5)

Also check that every task has the same number of attempts, that each result is a boolean produced by the evaluator, and that retries are not accidentally counted as independent attempts. If you use random seeds, record them. If the agent uses a live service or changing database, document the isolation strategy because the attempts may not be comparable.

Failure cases and interpretation

Correlated attempts

The equation passk = pk assumes independent attempts. Real agent failures are often correlated: a malformed tool schema, a broken prompt, or an unavailable dependency can cause every retry to fail. In that situation, the observed passk remains the useful empirical metric, but pk is not a valid prediction model.

Changing task state

If the first attempt creates a file, sends a message, or modifies a database row, later attempts may be testing a different task. Use resettable fixtures or define the state transition explicitly. Otherwise, label the result as a stateful workflow measurement rather than repeatability on a fixed task.

Evaluator errors

A weak evaluator can make any metric look precise. Prefer deterministic checks where possible: schema validation, exact state assertions, test suites, or invariant checks. For subjective outputs, use a documented rubric and report evaluator agreement or uncertainty when available.

Small samples

With a small task set, passk changes in large steps. If 10 tasks are evaluated, one task changes the metric by 0.1. Report the numerator and denominator—such as “7 of 20 tasks”—instead of presenting only a rounded percentage.

Selection bias

Do not remove difficult tasks after inspecting failures. Freeze the task list before the measurement, or clearly separate exploratory analysis from the final evaluation set.

What to report

A useful reliability report makes the measurement reproducible without exposing secrets. Include:

  • the task-set definition and number of tasks;
  • the repetition counts evaluated;
  • pass1 and passk with raw counts;
  • pass@k only when candidate selection is part of the workflow;
  • model, prompt, tool, evaluator, and environment versions;
  • the reset, seed, timeout, and retry policies;
  • known exclusions and failure categories.

Do not describe passk as a universal probability that the agent will succeed in the future. It is an estimate for the defined task distribution and evaluation setup. Its value is that it matches the operational question more closely than a single attempt when every recurring execution must work.

Practical takeaway

Start with the metric that matches the product behavior. If the system may try several candidates and keep one successful result, measure pass@k. If the system must execute a recurring action successfully every time, measure passk.

The smallest useful experiment is straightforward: freeze a task set, run each task five times in an isolated environment, record evaluator results, and publish pass1 alongside pass5. The difference between those two numbers reveals how much a one-shot success rate is hiding.

Continue with the guides or look up related terms in the glossary.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →