Evaluation · Prompt engineering
Prompt Ensembles: When Five Variants Beat One

Five model calls can look like a reliability strategy. They can also be five times the cost with almost no additional signal. The difference is usually explained by error correlation, aggregation, and how ties are resolved.
Prompt ensembles are useful only when their members fail differently enough for the aggregator to recover from individual mistakes. This article gives you a reproducible test: hold the dataset and model configuration fixed, compare a single prompt with self-consistency and deliberately different prompt variants, then report accuracy together with calls, latency, cost, disagreement, and failure modes.
The question to answer
The practical question is not “does asking five times produce a better answer?” It is:
Does the additional accuracy from five coordinated predictions justify the additional calls under the same quality, latency, and budget constraints?
That wording changes the experiment. A five-call system that improves accuracy by one percentage point may be a bad choice for an interactive product. A smaller improvement may be valuable in an offline, high-consequence workflow. The decision depends on the operating point, not on the ensemble score alone.
Keep three quantities separate:
- Quality: how often the final prediction is correct, and how severe its errors are.
- Resource use: model calls, input and output tokens, latency, and monetary cost.
- Reliability: whether gains hold across slices, random seeds, prompt variants, and evaluation runs.
What counts as an ensemble?
An ensemble combines multiple predictions for the same input. The members may differ in wording, decomposition, examples, role framing, decoding settings, or even model family. The final answer is produced by an aggregation rule, such as majority vote or a confidence-weighted decision.
There are three comparisons worth making:
| System | What changes | What it tests |
|---|---|---|
| Single prompt | One fixed prompt and one call | Baseline quality and cost |
| Self-consistency | Same prompt repeated with permitted sampling variation | Whether independent-looking samples help |
| Prompt ensemble | Several fixed prompt variants | Whether prompt diversity creates useful disagreement |
Self-consistency is not automatically diverse. If decoding is deterministic, repeated calls may be identical. If the model uses the same reasoning path, a repeated error can appear five times. Conversely, changing prompts can increase variation without increasing correctness. The experiment must measure both.
A concrete case: classification with five labels
Assume you classify support messages into five known labels. Each example has one adjudicated reference label. The production system currently uses one prompt and returns one label. You want to know whether five prompt variants improve classification enough to justify their cost.
Define the task before writing variants:
Input: one support message
Output: exactly one label from:
- billing
- access
- bug
- cancellation
- other
Evaluation: exact match against the reference label
Do not let the ensemble introduce a new task. Every member should receive the same input, label set, output contract, and relevant policy context. If one variant is allowed to use extra information, it is no longer a fair prompt-only comparison.
Design five variants with different failure risks
The objective is not to make five prompts sound different. It is to create plausible decision procedures whose errors are not perfectly correlated.
- Direct classification: choose the best label from the message.
- Definition-first: restate the meaning of each label, then classify.
- Contrastive: compare the two most plausible labels and explain the distinction internally.
- Evidence-first: identify the decisive phrase before selecting a label.
- Adversarial: check whether the first choice could be confused with a neighboring label.
Keep the output schema identical. If explanations are needed for audit, store them separately from the final label. Do not evaluate one member on a richer output format unless the production system will also use that format.
Here is a configuration shape that makes the experiment explicit:
{
"task": "support_message_classification",
"labels": ["billing", "access", "bug", "cancellation", "other"],
"members": [
{"id": "direct", "prompt_version": "v1"},
{"id": "definitions", "prompt_version": "v2"},
{"id": "contrastive", "prompt_version": "v3"},
{"id": "evidence_first", "prompt_version": "v4"},
{"id": "adversarial", "prompt_version": "v5"}
],
"aggregation": {
"method": "majority_vote",
"tie_break": "single_prompt"
}
}
Control the experiment
Use a fixed evaluation set that was not used to invent the variants. Record the dataset version, prompt text, model identifier, decoding parameters, tool availability, system instructions, and parser version. If any of these change between systems, record the change and treat the result as a different experiment.
A minimal run table should contain one row per example and member:
example_id,member_id,prediction,valid_format,input_tokens,
output_tokens,latency_ms,request_status,error_message
For every example, also store the reference label and the final ensemble prediction. Never overwrite raw member predictions with the aggregate. The raw predictions are required to measure agreement and error correlation.
Compare aggregation rules
Start with simple rules. Complexity can hide where the gain came from.
Majority vote
For five members, the majority label is the label receiving at least three votes. This rule is easy to audit and does not require calibrated confidence. Its weakness is that five correlated members can confidently agree on the same wrong label.
Weighted vote
Assign each member a weight based only on training or validation data, never on the held-out test labels. A weighted vote can help when one prompt is consistently stronger, but it can also overfit a small validation set.
score(label) = sum(weight(member) for member
if member_prediction == label)
Abstention or escalation
If no label reaches a required threshold, send the example to a fallback process. This changes the metric: report coverage and accuracy separately. An ensemble that is highly accurate only when it answers may be useful, but it is not equivalent to a system that always predicts.
Tie-break policies are part of the model
A five-member ensemble can tie when there are multiple labels or when members fail to return valid predictions. The tie-break policy must be defined before evaluation.
Reasonable policies include:
- Use the single-prompt baseline.
- Use the highest validation-performing member.
- Run one additional adjudicator call.
- Abstain or escalate.
- Choose a deterministic label priority, if the domain explicitly requires it.
Each policy has a different cost and risk profile. An extra adjudicator call makes the system a six-call or conditionally six-call system. A fixed label priority may silently favor one class. Abstention improves safety in some settings while reducing coverage. Report the policy as part of the configuration, not as an implementation footnote.
Measure error correlation, not just agreement
Agreement measures whether members produce the same prediction. Error correlation asks whether they are wrong on the same examples. These are related but not identical.
For each pair of members, create an error indicator:
error(member, example) =
1 if prediction != reference_label
0 otherwise
Then compare the error vectors across examples. You can use a correlation coefficient, Jaccard similarity of error sets, or simply count shared errors. The simplest diagnostic is often the most interpretable:
shared_errors(A, B) =
count(example where error(A, example) == 1
and error(B, example) == 1)
Also report pairwise disagreement:
disagreement(A, B) =
mean(prediction(A, example) != prediction(B, example))
High disagreement is not automatically good. It may indicate useful diversity, unstable parsing, or random guessing. The useful pattern is disagreement concentrated on cases where at least one member is correct and the aggregator selects that correct answer.
Use a cost-adjusted comparison
Let:
Abe accuracy.Cbe total cost or a chosen resource measure.Lbe latency at the percentile relevant to your product.Nbe the number of evaluated examples.
Report absolute and relative changes:
accuracy_gain = A_ensemble - A_baseline
extra_cost = C_ensemble - C_baseline
gain_per_extra_cost = accuracy_gain / extra_cost
Do not treat gain per dollar as a universal score. It is a reporting aid. A product decision may instead require:
accuracy_ensemble >= required_accuracy
latency_p95_ensemble <= latency_budget
cost_per_example_ensemble <= cost_budget
For a fair comparison, calculate cost from actual recorded usage where possible. If provider pricing is not available or changes over time, report tokens and calls without inventing a currency amount.
A reproducible evaluation procedure
- Freeze a labeled evaluation set and assign it a version identifier.
- Freeze the model configuration, prompts, parser, and aggregation code.
- Run the single-prompt baseline.
- Run repeated calls with the same prompt if self-consistency is in scope.
- Run all five prompt variants on every example.
- Save raw outputs, parsed predictions, failures, tokens, and latency.
- Apply each aggregation rule to the same raw predictions.
- Apply each tie-break policy and record any additional calls.
- Compute accuracy, per-label metrics, disagreement, shared errors, coverage, latency, and resource use.
- Repeat the evaluation with a predefined random seed policy or repeated samples.
- Inspect examples where the ensemble differs from the baseline.
- Make the decision only after checking confidence intervals or an equivalent uncertainty analysis.
The last step matters. A small accuracy difference on a modest test set may be noise. Use a paired analysis because every system sees the same examples. For classification, compare the examples where the baseline and ensemble disagree, then examine which system is correct. A bootstrap over examples is a practical option when its assumptions and resampling procedure are documented.
Minimal pseudocode
for example in evaluation_set:
predictions = []
for member in members:
raw = call_model(
prompt=member.prompt,
input=example.text,
config=frozen_model_config
)
parsed = parse_label(raw, allowed_labels)
save_raw_result(example.id, member.id, raw, parsed)
if parsed is not None:
predictions.append((member.id, parsed))
final_label = aggregate(
predictions=predictions,
rule=aggregation_rule,
tie_break=tie_break_policy
)
save_final_result(example.id, final_label, predictions)
metrics = evaluate_saved_results(
references=evaluation_set.labels,
predictions=saved_final_results
)
This is a test outline, not a reported result. It deliberately leaves the model client, parser, retry behavior, and storage implementation open because those depend on the system being evaluated.
Failure cases to test explicitly
Correlated prompt errors
All variants may misread the same ambiguous phrase or inherit the same flawed label definition. Majority vote cannot fix a shared blind spot. Review shared-error clusters, not only aggregate accuracy.
Invalid or incomplete outputs
One member may return an explanation instead of a label, a label outside the allowed set, or no output after a timeout. Define whether retries count as additional calls and whether invalid members are removed or treated as abstentions.
Prompt leakage
A variant may accidentally include an answer pattern, test-set wording, or metadata unavailable in production. Keep prompt construction versioned and review the rendered requests.
Unequal information
One prompt may ask for a tool lookup or longer context while another receives only the user message. That is a system comparison, not an isolated prompt-ensemble comparison.
Parser-induced gains
An ensemble can appear better because its outputs are easier for the parser to recognize. Evaluate raw semantic correctness and format validity separately.
Selective reporting
Do not report only the best aggregation rule, seed, or slice. Predefine the primary metric and publish the full comparison table, including rules that performed worse.
What a useful result table looks like
Fill this table from your run rather than copying a headline number:
| System | Calls/example | Accuracy | Coverage | p95 latency | Tokens/example | Notes |
|---|---|---|---|---|---|---|
| Single prompt | Measured | Measured | Measured | Measured | Measured | Baseline |
| Self-consistency | Measured | Measured | Measured | Measured | Measured | Same prompt, sampled calls |
| Five-variant majority | Measured | Measured | Measured | Measured | Measured | Fixed tie-break |
| Five-variant weighted | Measured | Measured | Measured | Measured | Measured | Weights frozen before test |
How to decide whether five beats one
Choose the ensemble only when the measured gain survives the following checks:
- The gain is visible on a held-out set and not only on prompt-development examples.
- The gain persists across meaningful data slices.
- At least some member errors are complementary rather than identical.
- The aggregation rule and tie-break policy are fixed before the final comparison.
- Latency, calls, tokens, and operational complexity fit the product budget.
- The ensemble does not hide a dangerous increase in confident errors.
If the five-member system is more accurate but too slow, consider conditional routing: call one member first and invoke additional members only when a confidence or disagreement signal warrants it. Evaluate that router as a new system, including the cases where it decides not to spend more calls.
Limitations
This protocol does not prove that prompt diversity will generalize to a different model, dataset, language, or task. Error correlation is dataset-dependent. A prompt that is complementary on one domain may be redundant on another. Aggregation can also fail when labels are ambiguous, references are inconsistent, or the task requires a structured explanation rather than a single discrete answer.
Accuracy alone is insufficient for high-impact decisions. Add domain-specific harm metrics, calibration checks, human review, and abstention analysis where appropriate. Do not infer safety from agreement.
Bottom line
Five variants beat one when they provide recoverable diversity: they disagree on cases where at least one member has useful evidence, and the aggregation policy selects that evidence reliably. The only defensible way to find out is to measure the full trade-off—quality, shared errors, calls, tokens, latency, failures, and tie behavior—on a frozen evaluation set.
For adjacent methods and definitions, continue with the evaluation guides or browse the glossary.