Cutting agent costs with trace-based model routing

What you will build

You will build a two-route policy for an AI agent:

• cheap sends an operation to the less expensive model;
• strong sends it to the more capable model;
• a failed cheap attempt may escalate to strong, but that escalation remains visible in the report.

The router does not claim to measure intelligence or universal prompt difficulty. It predicts a narrower event: whether the cheap route is likely to satisfy this application’s acceptance contract, with the current instructions, tools, validators, and input distribution.

The completed work should leave six reproducible artifacts:

1. a privacy-reviewed dataset derived from production-like traces;
2. paired cheap and strong outcomes for the same operations;
3. a versioned router with a calibrated decision threshold;
4. a shadow-mode comparison against fixed routing policies;
5. a canary rollout with an immediate strong-only fallback;
6. a report covering cost, quality, latency, routing errors, escalations, and retries.

Concrete case: an internal support agent

Use an internal product-support agent as the working example. This is a reproducible design scenario, not a claim about a customer or a previously measured deployment.

The agent performs four operations:

1. answers questions from an approved knowledge base;
2. extracts required fields from a support request;
3. classifies the request and proposes the next action;
4. uses a read-only tool to retrieve ticket status.

The cheap model may be sufficient for direct retrieval, short classification, and constrained extraction. The strong model may be necessary when the context is contradictory, the policy is ambiguous, the conversation is long, or several dependent tool calls are required.

Both routes must receive equivalent inputs: the same normalized conversation, the same retrieved documents, the same prompt version, the same tool permissions, and the same output contract. If adapters differ because providers expose different interfaces, record the adapter version as part of the route.

The routing unit is one completed business operation, not one model call. An operation starts when the application accepts a user request and ends when it returns an accepted result, safely refuses, reaches a terminal error, or exhausts its retry budget.

1. Define acceptance before optimizing cost

Before collecting labels, write an operation contract. “The request returned HTTP 200” is not a quality criterion. The contract must describe an outcome the product can accept.

operation: support_answer:v4

starts_when:
  - user_message_accepted

passes_when:
  - output_schema_valid
  - policy_checks_pass
  - required_claims_supported_by_supplied_sources
  - requested_read_action_completed_or_safely_declined
  - no_critical_violation
  - no_corrective_user_retry_inside_evaluation_window

fails_when:
  - deterministic_validator_rejects
  - expert_review_rejects
  - unauthorized_action_attempted
  - retry_budget_exhausted
  - operation_deadline_exceeded

quality_floor:
  critical_violation_rate: 0
  accepted_operation_rate: PRODUCT_DEFINED_VALUE

latency_budget_ms: PRODUCT_DEFINED_VALUE
user_retry_window: PRODUCT_DEFINED_VALUE

Replace every PRODUCT_DEFINED_VALUE with an approved threshold. There is no universal acceptable error rate. A weak draft title and an incorrect account-status action do not carry the same risk.

Use three layers of evaluation:

1. Deterministic checks: schema validity, required fields, tool allowlists, argument constraints, supported citations, and prohibited values.
2. Expert checks: factual correctness, policy interpretation, sufficient explanation, and appropriate refusal.
3. Behavioral checks: a corrective user message, a human takeover, a repeated operation, or successful task completion.

Name each label according to what it proves. A JSON validator proves schema_valid, not answer_correct. Combining weak signals under an ambitious label is an easy way to train a confidently wrong router.

2. Establish policies worth comparing

Evaluate at least three policies on the same eligible operations:

A practical rule baseline can route operations with tools, large contexts, previous failures, or high-risk action types directly to strong. A learned router is justified only if it produces a better cost-risk trade-off than this understandable baseline.

3. Instrument the complete operation trace

A trace must connect the user request, routing decision, every attempt, every model call, validator results, tool activity, and the terminal outcome. This is the minimum observability needed to distinguish genuine savings from hidden retries.

{
  "trace_id": "generated",
  "operation_id": "generated",
  "operation_type": "support_answer:v4",
  "cohort": "eligible_support",
  "policy_version": "strong_only:v1",
  "router_version": null,
  "feature_version": "features:v1",
  "prompt_version": "support:v4",
  "selected_route": "strong",
  "router_score": null,
  "decision_reason": "baseline",
  "started_at": "ISO-8601",
  "completed_at": "ISO-8601",
  "terminal_status": "accepted",
  "quality_pass": true,
  "critical_violation": false,
  "user_retry": false,
  "retry_of_operation_id": null
}

Record each model call separately. At minimum, include:

• call_id, operation_id, and attempt number;
• provider, actual model identifier, route, and adapter version;
• input, cached-input, reasoning, and output token counts when exposed;
• confirmed charge, currency, and pricing-version identifier;
• queue time, time to first token, and total call latency;
• schema, policy, factual, and tool-result checks;
• error class, automatic repair, retry, and escalation status.

CREATE TABLE router_operations (
  operation_id        TEXT PRIMARY KEY,
  trace_id            TEXT NOT NULL,
  operation_type      TEXT NOT NULL,
  cohort              TEXT NOT NULL,
  policy_version      TEXT NOT NULL,
  router_version      TEXT,
  feature_version     TEXT NOT NULL,
  selected_route      TEXT NOT NULL,
  router_score        DOUBLE PRECISION,
  decision_reason     TEXT NOT NULL,
  started_at          TIMESTAMP NOT NULL,
  completed_at        TIMESTAMP,
  terminal_status     TEXT,
  quality_pass        BOOLEAN,
  critical_violation BOOLEAN,
  user_retry          BOOLEAN NOT NULL DEFAULT FALSE,
  retry_of            TEXT
);

CREATE TABLE router_calls (
  call_id             TEXT PRIMARY KEY,
  operation_id        TEXT NOT NULL,
  attempt_no          INTEGER NOT NULL,
  route               TEXT NOT NULL,
  model_id            TEXT NOT NULL,
  prompt_version      TEXT NOT NULL,
  adapter_version     TEXT NOT NULL,
  input_tokens        INTEGER,
  cached_input_tokens INTEGER,
  output_tokens       INTEGER,
  charged_cost        NUMERIC,
  currency            TEXT,
  price_version       TEXT,
  queue_ms             INTEGER,
  first_token_ms       INTEGER,
  latency_ms           INTEGER,
  schema_valid        BOOLEAN,
  validator_pass      BOOLEAN,
  error_class         TEXT,
  escalated            BOOLEAN NOT NULL DEFAULT FALSE,
  created_at           TIMESTAMP NOT NULL,
  FOREIGN KEY (operation_id) REFERENCES router_operations(operation_id)
);

CREATE INDEX router_calls_by_operation
ON router_calls(operation_id, attempt_no);

Store unknown charges as NULL, never zero. A zero means “confirmed free”; NULL means “not reconciled.” Keep a separate estimated-cost column if estimates are operationally useful.

4. Turn traces into a privacy-reviewed dataset

Production traces can contain personal data, secrets, internal URLs, document contents, and tool results. Do not export raw telemetry into a training directory by default.

1. Define the permitted operation types and time range.
2. Exclude opted-out, regulated, or legally restricted records.
3. Replace stable user and tenant identifiers with scoped pseudonyms.
4. Remove credentials, authorization headers, cookies, signed URLs, and tool secrets.
5. Preserve task structure: redaction must not erase the signals the router needs.
6. Record the redaction-policy version and dataset manifest.
7. Restrict retention and access independently from production logs.

{
  "dataset_id": "router-replay-YYYYMMDD",
  "source_window": {
    "from": "APPROVED_START",
    "to": "APPROVED_END"
  },
  "operation_types": ["support_answer:v4"],
  "redaction_policy": "router-redaction:v2",
  "prompt_version": "support:v4",
  "tool_contract_version": "support-read-tools:v3",
  "feature_version": "features:v1",
  "record_count": "COMPUTED_VALUE",
  "content_hash": "COMPUTED_VALUE"
}

Do not use raw text, email addresses, customer identifiers, or full URLs as metric labels. High-cardinality telemetry is expensive, hard to aggregate, and prone to leaking sensitive information.

5. Sample operations that represent the application

A random sample of successful calls is insufficient. It excludes the very cases that determine whether routing is safe.

Stratify the sample by variables known before routing:

• operation type and product surface;
• input-length and context-length buckets;
• language, attachment presence, and retrieval use;
• number and type of available tools;
• new conversation versus follow-up;
• previous attempt state;
• historical terminal outcome;
• time period, including recent traffic.

Deliberately include failures, timeouts, repairs, human escalations, and user retries. Weight the final aggregate back to the real traffic distribution; otherwise, oversampling rare failures will distort the cost projection.

Keep an untouched, recent holdout period. Model behavior, prompts, retrieval indexes, and traffic change over time, so a purely random split can make the test look easier than deployment.

6. Run paired counterfactual replay

In an isolated benchmark, execute each eligible saved input against both routes. Pairing matters: comparing cheap calls from Monday with strong calls from Friday confounds route quality with traffic differences.

for example in evaluation_set:
    cheap = run_route(
        route="cheap",
        input=example.normalized_input,
        prompt_version=example.prompt_version,
        tools=read_only_replay_tools
    )

    strong = run_route(
        route="strong",
        input=example.normalized_input,
        prompt_version=example.prompt_version,
        tools=read_only_replay_tools
    )

    store_pair(
        example_id=example.id,
        cheap=cheap,
        strong=strong,
        evaluator_version="acceptance:v3"
    )

Replay tools must not change live systems. Replace writes with recorded fixtures or a deterministic simulator. If a historical read cannot be reconstructed, mark the example unevaluable rather than silently giving one route different evidence.

Control nondeterminism:

• pin model, prompt, adapter, tool contract, and evaluator versions;
• use the same retrieved evidence for both routes;
• record sampling parameters and provider request identifiers;
• repeat a subset to measure outcome variance;
• keep transient infrastructure errors separate from answer-quality failures.

A strong-model response is not automatically ground truth. Both routes can fail. Apply the same acceptance process to both outputs.

7. Build the routing label

The simplest target is whether the cheap route passes the acceptance contract:

cheap_success =
    schema_valid
    AND policy_pass
    AND required_facts_supported
    AND tool_outcome_valid
    AND NOT critical_violation

For routing, paired results support four useful classes:

Keep infrastructure failures distinct from semantic failures. A provider timeout does not prove that the model was incapable of answering. Depending on the goal, either exclude these records from capability training or model availability as a separate routing signal.

Account for corrective user requests

A technically accepted response may still be wrong if the user immediately restates the request, corrects a field, or asks for the promised action again. Link likely retries using a conservative rule based on actor, operation type, time window, and semantic or business identifiers.

user_retry =
    same_scoped_actor
    AND same_operation_type
    AND within_configured_window
    AND (
      explicit_correction
      OR same_business_object
      OR reviewed_semantic_match
    )

Do not declare every follow-up a failure. A user may ask a legitimate next question. Report confirmed retries separately from inferred retries and manually review a sample of both.

8. Split data without leakage

Near-duplicate conversations commonly appear in support workloads. If one copy enters training and another enters testing, the reported performance will be optimistic.

1. Group exact duplicates by normalized-input hash.
2. Group related turns by conversation or case identifier.
3. Where appropriate, group repeated templates or the same business object.
4. Assign complete groups to train, validation, or test.
5. Reserve the newest time window as an additional temporal holdout.

train:
  time: earlier_window
  groups: assigned_train_groups

validation:
  time: earlier_window
  groups: assigned_validation_groups

test:
  time: later_untouched_window
  groups: assigned_test_groups

Fit text vocabularies, encoders, scalers, and missing-value rules on training data only. Do not inspect test errors repeatedly while adjusting features; that turns the test set into another validation set.

9. Use only features available before the decision

The router may use properties known before the first model call:

• operation type;
• input character or token-length bucket;
• conversation-turn count;
• retrieved-document count and aggregate length;
• attachment types;
• language category;
• tool availability and required action risk;
• whether structured output is required;
• previous attempt count;
• coarse policy or product area;
• privacy-safe text representation, if approved.

Do not use fields produced after routing, including cheap-model confidence, cheap output length, validator result, final latency, escalation status, or the user’s later correction. Those are future information and will not exist when the live router makes its choice.

{
  "operation_type": "support_answer",
  "input_token_bucket": "513-1024",
  "conversation_turn_bucket": "4-7",
  "retrieval_document_bucket": "1-3",
  "has_attachment": false,
  "requires_tool": true,
  "tool_risk": "read_only",
  "requires_structured_output": true,
  "attempt_bucket": "first",
  "language": "en"
}

Prefer stable, interpretable features first. A small router is easier to calibrate, debug, and run within a tight latency budget. Add embeddings or a compact language model only if simpler baselines leave a material, verified gap.

10. Train a calibrated probability model

A regularized logistic model or gradient-boosted tree is often an adequate first router. The desired output is an estimate of:

p_cheap_success = P(cheap route passes | pre-route features)

One possible configuration shape is:

router:
  objective: predict_cheap_success
  model_family: regularized_logistic_regression
  class_weight:
    cheap_success: 1
    cheap_failure_strong_success: PRODUCT_RISK_WEIGHT
  calibration:
    method: isotonic_or_sigmoid
    fit_on: validation_only
  versions:
    dataset: router-replay-YYYYMMDD
    features: features:v1
    evaluator: acceptance:v3

The risk weight is a product decision, not a value to copy from this article. Route errors that trigger an incorrect high-impact action should carry more weight than errors that merely require reformatting.

Measure discrimination and calibration separately. Ranking metrics show whether safer examples tend to receive higher scores. Calibration shows whether a score such as 0.90 corresponds to roughly that success frequency in the evaluated cohort. Threshold decisions depend on the latter.

Why accuracy is inadequate

If most operations are easy, a router that always chooses cheap can have high routing accuracy while still sending the small, costly failure class to the wrong model. Report at least:

• false-cheap rate: cheap selected where cheap fails and strong passes;
• false-strong rate: strong selected where cheap would pass;
• critical false-cheap count and rate;
• accepted-operation rate by route;
• calibration by score bucket;
• metrics by operation type, language, tool use, and length bucket.

11. Select the threshold using cost and risk

Do not select 0.5 merely because the router is a classifier. Evaluate candidate thresholds on validation data using complete operation economics.

if p_cheap_success >= threshold:
    route = "cheap"
else:
    route = "strong"

For each threshold, simulate:

expected_operation_cost =
    router_inference_cost
  + initial_model_cost
  + automatic_retry_cost
  + repair_cost
  + escalation_cost
  + evaluation_cost
  + configured_failure_penalty

subject to:
  accepted_operation_rate >= APPROVED_FLOOR
  critical_violation_rate <= APPROVED_LIMIT
  p95_operation_latency_ms <= APPROVED_BUDGET

Keep monetary spend and business-loss assumptions in separate columns. Provider charges can often be reconciled. Failure penalties are modeled assumptions and must be labeled as such.

Add an uncertainty zone

A practical policy can use two thresholds:

if safety_rule_requires_strong(features):
    route = "strong"
elif score >= cheap_threshold:
    route = "cheap"
elif score <= strong_threshold:
    route = "strong"
else:
    route = uncertainty_policy

The uncertainty policy may choose strong, apply a deterministic rule, request clarification, or use a separately tested intermediate route. Do not hide uncertainty by forcing every input onto cheap.

Apply hard safety rules first

Keep known high-risk categories out of the learned cheap route until they have dedicated evidence. Examples include write-capable tools, irreversible actions, authentication changes, money movement, legal commitments, and inputs outside the training distribution.

force_strong_when:
  - action_risk in ["write", "irreversible"]
  - operation_type not in trained_operation_types
  - required_feature_missing
  - router_unavailable
  - prompt_version not in compatible_prompt_versions
  - context_size outside_trained_range

12. Implement the runtime policy

def execute_operation(operation):
    features = build_features(operation)

    safety = evaluate_safety_rules(features)
    if safety.force_strong:
        decision = Decision(
            route="strong",
            score=None,
            reason=safety.reason
        )
    else:
        score = router.predict_proba(features)
        route = "cheap" if score >= CHEAP_THRESHOLD else "strong"
        decision = Decision(
            route=route,
            score=score,
            reason="trained_policy"
        )

    persist_decision_before_call(
        operation_id=operation.id,
        router_version=ROUTER_VERSION,
        feature_version=FEATURE_VERSION,
        decision=decision
    )

    first = call_route(
        operation=operation,
        route=decision.route,
        idempotency_key=operation.id + ":attempt:1"
    )

    checked = validate(first)

    if checked.pass_all:
        return finalize(operation, first, status="accepted")

    if decision.route == "cheap" and checked.escalation_allowed:
        second = call_route(
            operation=operation,
            route="strong",
            idempotency_key=operation.id + ":attempt:2"
        )
        return finalize_escalation(operation, first, second)

    return finalize(operation, first, status="rejected")

Persist the routing decision before the billable call. Otherwise, a worker crash can leave a charge with no recorded policy or score.

Use an idempotency key when the provider or your application boundary supports it. A retry caused by an uncertain network result must not accidentally create another tool action.

Separate these concepts:

• call retry: the same route is called again after a transient error;
• repair: another call corrects schema or formatting;
• escalation: a cheap result fails validation and the strong route is invoked;
• user retry: the user repeats or corrects the logical request.

All four consume resources, but they indicate different failures.

13. Start in shadow mode

In shadow mode, production continues using strong_only. The router computes a hypothetical decision but does not control the response.

{
  "effective_route": "strong",
  "effective_policy": "strong_only:v1",
  "shadow_route": "cheap",
  "shadow_router_version": "trace-router:v1",
  "shadow_score": 0.93,
  "shadow_reason": "trained_policy"
}

Shadow mode verifies feature availability, schema compatibility, score distributions, decision latency, version logging, and cohort assignment without exposing users to route errors.

It does not prove cheap-route quality unless the cheap route is also evaluated. You can obtain that evidence through offline paired replay or a separately approved shadow call. Extra shadow calls cost money and may process sensitive data, so include them in the experiment budget and privacy review.

Exit criteria

• no missing required features in the eligible cohort;
• router and feature versions present on every decision;
• decision latency within its allocated budget;
• score distribution consistent with offline observations or explained;
• no critical false-cheap cases in the reviewed release sample;
• cost projections include retries and escalations;
• strong-only fallback tested successfully.

14. Run a controlled canary

Enable the router only for an eligible low-risk cohort. Assignment must be stable at the appropriate unit—usually actor, conversation, or operation—so a user is not moved unpredictably between policies.

routing_experiment:
  eligibility:
    operation_types: ["support_answer:v4"]
    action_risk: ["read_only", "no_tool"]
    supported_languages: ["en"]
  assignment_unit: conversation_id
  policies:
    control: strong_only:v1
    treatment: trace_router:v1
  exclusions:
    - write_capable_tool
    - unresolved_previous_failure
    - unsupported_prompt_version
    - missing_required_feature
  emergency_fallback: strong_only:v1

Choose traffic exposure and duration from your power analysis, operational risk, and rollback capability. A percentage copied from another system has no evidential value.

Freeze the router, thresholds, prompt, evaluator, and cohort definition during the measurement window unless safety requires a change. If anything changes, start a new report segment rather than merging incompatible versions.

15. Compute operation-level metrics

Total cost

confirmed_model_cost =
  SUM(charged_cost WHERE charge_status = "confirmed")

complete_operation_cost =
    confirmed_model_cost
  + confirmed_router_cost
  + confirmed_evaluator_cost
  + confirmed_tool_cost

cost_per_started_operation =
  SUM(complete_operation_cost) / COUNT(started_operations)

cost_per_accepted_operation =
  SUM(complete_operation_cost) / COUNT(accepted_operations)

Report total spend and cost per accepted operation. A cheap policy can lower call cost while increasing cost per accepted result through repairs, escalations, and failures.

Show the share of costs still estimated or unknown. Do not merge currencies without a dated, documented conversion rule.

Quality

accepted_operation_rate =
  accepted_operations / eligible_started_operations

critical_violation_rate =
  critical_violations / evaluated_operations

review_acceptance_rate =
  reviewer_accepted / reviewer_evaluated

Include evaluator coverage. A high pass rate based on a small or selectively reviewed subset is not equivalent to broad evidence.

Latency

Measure end-to-end operation latency, not just model latency:

operation_latency_ms =
  terminal_timestamp - accepted_request_timestamp

Report median, p90, p95, and p99 where sample size permits. Separate queue time, router time, first call, validation, retry, and escalation so the tail has an explanation.

Wrong routes

false_cheap =
  selected_route = "cheap"
  AND cheap_acceptance = false
  AND strong_counterfactual_acceptance = true

false_strong =
  selected_route = "strong"
  AND cheap_counterfactual_acceptance = true

false_cheap_rate =
  false_cheap / evaluated_cheap_decisions

false_strong_rate =
  false_strong / evaluated_strong_decisions

Live traffic usually lacks both counterfactual outcomes. Estimate wrong-route rates from paired replay, approved exploration, or a reviewed sample. Label the source and coverage instead of presenting an incomplete denominator as universal truth.

Retries, repairs, and escalations

automatic_retry_rate =
  operations_with_call_retry / started_operations

repair_rate =
  operations_with_repair / started_operations

cheap_to_strong_escalation_rate =
  cheap_started_then_strong / cheap_started_operations

user_retry_rate =
  confirmed_or_inferred_user_retries / eligible_completed_operations

Also report average calls per operation and the cost of escalated operations. A high escalation rate can make routing look safe because the strong model repairs the output, while eliminating the expected savings.

16. Produce a report without invented results

Generate the table below from your warehouse or evaluation files. Leave missing values explicitly unavailable rather than filling them with assumptions.

Accompany the summary with these slices:

• operation type;
• input and context-length bucket;
• language;
• tool requirement and action risk;
• new conversation versus follow-up;
• router-score bucket;
• prompt, router, feature, adapter, and evaluator version;
• failure class.

Every report should state the evaluation window, cohort definition, exclusion count, sample size, evaluator coverage, price version, currency, unknown-charge count, and confidence interval or uncertainty method where appropriate.

Routing-error register

17. Verify the report with independent queries

Aggregate at operation level before comparing policies. Joining calls directly to evaluations can multiply rows and inflate cost.

WITH call_totals AS (
  SELECT
    operation_id,
    COUNT(*) AS call_count,
    SUM(CASE WHEN charged_cost IS NOT NULL
             THEN charged_cost ELSE 0 END) AS confirmed_model_cost,
    SUM(CASE WHEN charged_cost IS NULL
             THEN 1 ELSE 0 END) AS unknown_charge_calls,
    MAX(CASE WHEN escalated THEN 1 ELSE 0 END) AS had_escalation
  FROM router_calls
  GROUP BY operation_id
),
operation_facts AS (
  SELECT
    o.operation_id,
    o.policy_version,
    o.selected_route,
    o.quality_pass,
    o.critical_violation,
    o.user_retry,
    EXTRACT(EPOCH FROM (o.completed_at - o.started_at)) * 1000
      AS operation_latency_ms,
    COALESCE(c.call_count, 0) AS call_count,
    COALESCE(c.confirmed_model_cost, 0) AS confirmed_model_cost,
    COALESCE(c.unknown_charge_calls, 0) AS unknown_charge_calls,
    COALESCE(c.had_escalation, 0) AS had_escalation
  FROM router_operations o
  LEFT JOIN call_totals c USING (operation_id)
  WHERE o.cohort = 'eligible_support'
)
SELECT
  policy_version,
  COUNT(*) AS started_operations,
  SUM(CASE WHEN quality_pass THEN 1 ELSE 0 END) AS accepted_operations,
  SUM(CASE WHEN critical_violation THEN 1 ELSE 0 END)
    AS critical_violations,
  SUM(confirmed_model_cost) AS confirmed_model_cost,
  SUM(confirmed_model_cost)
    / NULLIF(SUM(CASE WHEN quality_pass THEN 1 ELSE 0 END), 0)
    AS model_cost_per_accepted_operation,
  AVG(call_count) AS calls_per_operation,
  AVG(had_escalation) AS escalation_rate,
  AVG(CASE WHEN user_retry THEN 1.0 ELSE 0.0 END) AS user_retry_rate,
  SUM(unknown_charge_calls) AS unknown_charge_calls
FROM operation_facts
GROUP BY policy_version
ORDER BY policy_version;

Check repeated calls independently:

SELECT
  operation_id,
  route,
  COUNT(*) AS calls,
  MIN(created_at) AS first_call,
  MAX(created_at) AS last_call,
  SUM(COALESCE(charged_cost, 0)) AS confirmed_cost
FROM router_calls
GROUP BY operation_id, route
HAVING COUNT(*) > 1
ORDER BY calls DESC;

Reconcile these aggregates with provider billing exports for the same model identifiers and time window. Explain differences caused by delayed charges, credits, cached-token accounting, failed calls, or timezone boundaries.

18. Verification before increasing traffic

1. Trace completeness: every billable call maps to one operation and one recorded decision.
2. Counterfactual comparability: paired routes used equivalent inputs, evidence, prompts, and tools.
3. Leakage prevention: no feature depends on the selected route’s output or later user behavior.
4. Split integrity: duplicate and related conversations do not cross dataset partitions.
5. Evaluator validity: deterministic and expert labels have documented coverage and versions.
6. Calibration: score buckets on the holdout set match observed cheap-route success closely enough for the decision policy.
7. Safety rules: unknown and high-risk operations default to strong.
8. Retry accounting: call retries, repairs, escalations, and user retries are counted separately.
9. Financial reconciliation: confirmed, estimated, and unknown costs are not mixed.
10. Latency accounting: the report uses end-to-end operation latency.
11. Version isolation: incompatible router, prompt, adapter, feature, or price versions are not aggregated.
12. Fallback: an immediate rollback to strong_only has been exercised.

Minimum automated tests

test_unknown_operation_routes_to_strong
test_missing_feature_routes_to_strong
test_router_timeout_routes_to_strong
test_write_tool_routes_to_strong
test_decision_is_persisted_before_model_call
test_retry_reuses_logical_operation_id
test_idempotency_key_changes_only_by_attempt
test_escalation_cost_is_included
test_unknown_charge_is_not_zero
test_future_features_are_rejected
test_duplicate_groups_do_not_cross_splits
test_report_aggregates_calls_before_joining
test_policy_versions_are_not_merged
test_strong_only_fallback_bypasses_router

Common failure cases

The strong model is treated as ground truth

The strong route can also hallucinate, misuse a tool, or violate policy. Evaluate both routes against the same contract and review disagreements.

Training uses future information

A feature such as validator outcome makes offline performance excellent and live routing impossible. Enforce a schema of fields available at decision time.

Training and test contain duplicate conversations

The router memorizes templates or cases. Group before splitting and maintain a recent temporal holdout.

The report counts calls instead of operations

A cheap attempt followed by a strong escalation appears as two successes or two observations. Aggregate the entire trace into one terminal operation first.

Escalation hides bad routing

Final quality remains acceptable because strong repairs cheap failures, but cost and tail latency increase. Report initial-route quality and escalation rate separately.

User follow-ups are all labeled as retries

Legitimate next questions become false failures. Use explicit correction signals, business identifiers, conservative matching, and reviewed samples.

Only averages are monitored

Average latency and quality can conceal a failing language, tool, or long-context segment. Inspect tails and cohort slices.

Current prices are applied to historical traffic

The apparent savings change when the price table changes. Store confirmed charges or a versioned price rule effective at call time.

The prompt changes without retraining or reevaluation

A prompt update can alter which operations the cheap model handles. Treat prompt compatibility as part of the router release.

Router failure defaults to cheap

This converts an infrastructure incident into a quality incident. For risk-sensitive operations, fail closed to the strong route.

Retries have no budget

A route that looks cheap per call can become expensive through repeated repairs. Cap attempts per operation and stop unchanged retries on permanent errors.

Limitations

A trace-trained router learns the distribution represented in its dataset. It may not generalize to a new product surface, language, prompt, retrieval index, tool, or model revision.

Counterfactual replay is imperfect when external state cannot be reconstructed. Live user reactions are also unavailable for hypothetical responses, so behavioral quality requires careful sampling or controlled deployment.

Evaluator errors become label errors. Deterministic checks are narrow, expert review is costly, and model-based judging requires its own validation. No single evaluator should silently define overall quality.

Provider behavior and prices can change. Recalculate economics with versioned charges and reevaluate route capability after material model changes.

A two-route policy does not solve every workload. Some operations need retrieval repair, clarification, a deterministic program, a human reviewer, or refusal rather than a stronger model.

Finally, routing itself adds operational complexity: another model or ruleset, feature computation, monitoring, version compatibility, and failure handling. If traffic is small or the models have similar economics, that complexity may cost more than it saves.

Operating the router after launch

The router becomes part of the production workflow. Monitor it as a versioned decision service, not as a static experiment.

• track input-feature and score-distribution drift;
• compare accepted-operation, escalation, and retry rates by route;
• review false-cheap cases and critical failures immediately;
• replay a stable regression set after model, prompt, tool, or evaluator changes;
• sample live operations for expert review;
• reconcile charges on a regular schedule;
• expire router versions that are incompatible with current prompts or features;
• retain a tested strong-only fallback.

Retrain on a declared trigger—material drift, a changed model, a changed prompt, enough newly reviewed errors, or an agreed schedule. Automatic retraining without gated evaluation can turn temporary anomalies into production policy.

Final checklist

• The routing unit is a completed business operation.
• Acceptance criteria exist before model comparison.
• Cheap and strong routes are evaluated on paired inputs.
• Both routes use equivalent evidence and permissions.
• Training data is privacy-reviewed and versioned.
• Related traces remain in one dataset partition.
• Features exist before the route decision.
• The router outputs a calibrated score.
• The threshold reflects cost and asymmetric risk.
• Hard safety rules override learned routing.
• Unknown inputs and router failures default safely.
• Shadow mode validates integration before control.
• The canary has stable assignment and explicit eligibility.
• Cost includes retries, repairs, evaluations, and escalations.
• Quality includes deterministic, expert, and behavioral evidence.
• Latency is measured end to end and includes tail percentiles.
• False-cheap and false-strong routes are reported separately.
• Automatic and user retries have distinct definitions.
• Unknown financial data remains unknown.
• The report records every relevant version and denominator.
• Rollback to strong_only is tested.

The router is ready only when it reduces verified cost per accepted operation without crossing the approved quality and latency limits. A lower model bill alone is not success if difficult answers fail, users repeat themselves, or cheap attempts routinely escalate to the strong route.