LLM Function Calling: Comparing Schemas and Tool Selection Across Providers

Why identical tools behave differently

Function calling is a protocol in which a model requests that the host application execute a named operation with structured arguments. The model does not run the function itself. It produces a proposed call; your application validates it, performs the operation, and may return a result to the model.

In practice, llm function calling is not one universal wire format. Providers differ in request fields, supported JSON Schema features, tool-choice controls, response envelopes, streaming events, parallel-call behavior, and the messages required to return a tool result. Models add another layer of variation: they interpret descriptions and ambiguous user requests differently.

The surrounding llm api also matters. A provider may reject an unsupported schema keyword before inference, silently ignore it, or accept the schema while the model later violates it. Those outcomes should not be collapsed into a single “function calling failed” metric.

A useful comparison separates four stages:

1. Request acceptance: did the provider accept the tool schema?
2. Tool decision: did the model call, abstain, or choose the wrong tool?
3. Argument construction: were the supplied values structurally and semantically correct?
4. Recovery: after a validation or execution error, did the model repair the call appropriately?

The concrete case: a support operations assistant

We will test three read-only tools for a fictional support workflow. No real customer data, credentials, or measured provider results are included. The benchmark produces its own evidence when you run it against the models you are authorized to use.

{
  "tools": [
    {
      "name": "lookup_order",
      "description": "Find one order by its exact public order ID.",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": {
            "type": "string",
            "pattern": "^ORD-[0-9]{6}$",
            "description": "Public order ID, for example ORD-104821."
          }
        },
        "required": ["order_id"],
        "additionalProperties": false
      }
    },
    {
      "name": "search_orders",
      "description": "Search orders when an exact order ID is not available.",
      "parameters": {
        "type": "object",
        "properties": {
          "email": {
            "type": "string",
            "description": "Customer email address."
          },
          "status": {
            "type": "string",
            "enum": ["processing", "shipped", "delivered", "cancelled"]
          },
          "limit": {
            "type": "integer",
            "minimum": 1,
            "maximum": 20,
            "default": 5
          }
        },
        "required": ["email"],
        "additionalProperties": false
      }
    },
    {
      "name": "get_shipping_quote",
      "description": "Estimate shipping price and delivery window for a destination.",
      "parameters": {
        "type": "object",
        "properties": {
          "country_code": {
            "type": "string",
            "pattern": "^[A-Z]{2}$",
            "description": "ISO-style two-letter uppercase country code."
          },
          "postal_code": {
            "type": "string",
            "description": "Postal code as user-provided text; preserve leading zeros."
          },
          "weight_grams": {
            "type": "integer",
            "minimum": 1,
            "maximum": 50000
          },
          "service": {
            "type": "string",
            "enum": ["standard", "express"]
          }
        },
        "required": ["country_code", "postal_code", "weight_grams"],
        "additionalProperties": false
      }
    }
  ]
}

This tool set exposes common problems without requiring destructive actions: overlapping lookup and search functions, formatted identifiers, enums, bounded integers, optional defaults, uppercase normalization, and postal codes that must remain strings.

The distinction between lookup_order and search_orders is especially valuable. A model that merely notices the word “order” cannot pass consistently; it must decide whether the user supplied an exact identifier.

Define a provider-neutral contract first

Do not begin by copying one provider’s request object throughout the application. Define a small internal representation, then translate it at the boundary of each llm api. This gives the benchmark one source of truth.

export type CanonicalTool = {
  name: string;
  description: string;
  parameters: Record<string, unknown>;
};

export type CanonicalToolCall = {
  id: string | null;
  name: string;
  arguments: unknown;
};

export type CanonicalOutcome =
  | { kind: "tool_calls"; calls: CanonicalToolCall[] }
  | { kind: "text"; text: string }
  | { kind: "refusal"; text: string | null }
  | { kind: "provider_error"; code: string; message: string };

Keep parsed arguments as unknown until validation. If an adapter casts "750" to 750, removes an unexpected property, or supplies a default before scoring, it hides the model’s actual output.

Use the schema intersection for the main comparison

Providers support different schema subsets. Start the primary benchmark with conservative constructs: object, string, integer, number, boolean, array, properties, required, enum, and simple bounds. Keep advanced keywords in a separate compatibility suite.

Maintain two schema files:

• tools.core.json contains the portable schema used to compare models.
• tools.compatibility.json probes keywords such as patterns, unions, nested constraints, and strictness controls.

If a provider cannot accept pattern, for example, remove it only in that provider’s compatibility adapter and record the transformation. Do not then claim that all models received semantically identical constraints.

Freeze the instructions

You are a support operations assistant.

Use a tool only when the user's request requires information supplied by that tool.
Never invent missing required values.
If exactly one required value is missing, ask one concise clarification question.
Use lookup_order only when an exact public order ID is present.
Use search_orders when the user supplies an email address but no exact order ID.
Do not call tools for greetings, explanations of policy, or unsupported actions.

Store this text in a versioned fixture. Whitespace changes, extra examples, and provider-specific instructions are experimental variables, not harmless edits.

Create a diagnostic test dataset

A test row needs more than an expected function name. It should state whether a call is expected, the exact acceptable arguments, and why alternative behavior is wrong.

{"id":"select-lookup-01","prompt":"Where is order ORD-104821?","expected":{"kind":"tool","name":"lookup_order","arguments":{"order_id":"ORD-104821"}},"tags":["selection","exact-id"]}
{"id":"select-search-01","prompt":"Show my shipped orders. My email is sam@example.test.","expected":{"kind":"tool","name":"search_orders","arguments":{"email":"sam@example.test","status":"shipped"}},"tags":["selection","enum"]}
{"id":"abstain-01","prompt":"Hello! What can you help me with?","expected":{"kind":"text"},"tags":["abstention"]}
{"id":"clarify-01","prompt":"How much is express shipping to Germany for 750 grams?","expected":{"kind":"clarification","missing":["postal_code"]},"tags":["missing-required"]}
{"id":"types-01","prompt":"Quote standard shipping for 750 grams to US postal code 02139.","expected":{"kind":"tool","name":"get_shipping_quote","arguments":{"country_code":"US","postal_code":"02139","weight_grams":750,"service":"standard"}},"tags":["types","leading-zero"]}
{"id":"normalize-01","prompt":"Shipping quote for 1 kg to ca, M5V 3L9.","expected":{"kind":"tool","name":"get_shipping_quote","arguments":{"country_code":"CA","postal_code":"M5V 3L9","weight_grams":1000}},"tags":["normalization","unit-conversion"]}
{"id":"unsupported-01","prompt":"Cancel every order on my account.","expected":{"kind":"text"},"tags":["unsupported","safety"]}
{"id":"conflict-01","prompt":"Find ORD-104821. My email is sam@example.test.","expected":{"kind":"tool","name":"lookup_order","arguments":{"order_id":"ORD-104821"}},"tags":["selection","overlap"]}

Use a reserved domain such as example.test and synthetic identifiers. The fixture does not need a live order database because the first evaluation stage stops after the proposed call.

Cover five behavior classes

1. Positive selection: the request clearly maps to exactly one function.
2. Confusable selection: two tools share vocabulary, but only one satisfies the request.
3. Abstention: no tool is necessary or available.
4. Argument pressure: values require conversion, normalization, or preservation of string formatting.
5. Missing information: the model must ask instead of guessing a required value.

Add paraphrases, but keep each original prompt. Replacing cases over time makes historical comparisons impossible. A practical convention is dataset_version: "1.0.0" in every run manifest.

Normalize provider requests and responses

Each provider adapter should have three responsibilities:

1. Translate canonical tools and messages into the provider request.
2. Extract text, refusals, errors, and every proposed tool call.
3. Preserve the unmodified request and response for diagnosis.

Keep the rest of the runner provider-agnostic. The following interface is enough for a first benchmark:

export interface ModelAdapter {
  provider: string;
  model: string;

  run(input: {
    system: string;
    prompt: string;
    tools: CanonicalTool[];
    toolChoice: "auto" | "required" | "none";
    temperature?: number;
  }): Promise<{
    outcome: CanonicalOutcome;
    rawRequest: unknown;
    rawResponse: unknown;
    latencyMs: number;
    usage?: {
      inputTokens?: number;
      outputTokens?: number;
    };
  }>;
}

Do not compare provider controls as if they were identical

A mode named auto can allow either text or a call. A mode named required may force any tool, a particular tool, or at least one tool, depending on the provider. Document the actual semantics beside every adapter.

Run the main selection test with the closest available equivalent of auto. Run forced-tool behavior as a separate experiment. Otherwise, a provider with a forced call may appear to have better recall while producing unusable calls on abstention cases.

A note on function calling openai integrations

When implementing function calling openai support, isolate the current request and response translation in its own adapter. Do not make the canonical format depend on provider-specific message roles, call IDs, strictness switches, or streaming event names. Apply the same rule to every other provider. This article intentionally avoids fixed endpoint and model identifiers because those operational details can change; pin the exact values available in your environment inside the run manifest.

Normalize calls without repairing them

export function normalizeArguments(value: unknown): unknown {
  if (typeof value !== "string") return value;

  try {
    return JSON.parse(value);
  } catch {
    return {
      __parse_error: true,
      __raw: value
    };
  }
}

Parsing a JSON string is protocol normalization. Converting units, fixing enum spelling, removing unknown keys, or changing types is semantic repair. Score those repairs separately if your production stack performs them.

Build the benchmark runner

The runner executes every case against every configured model, repeats cases, writes append-only records, and never calls a real business tool during the selection stage.

type TestCase = {
  id: string;
  prompt: string;
  expected:
    | { kind: "tool"; name: string; arguments: Record<string, unknown> }
    | { kind: "text" }
    | { kind: "clarification"; missing: string[] };
  tags: string[];
};

for (const adapter of adapters) {
  for (const testCase of cases) {
    for (let repeat = 0; repeat < config.repeats; repeat++) {
      const startedAt = new Date().toISOString();

      try {
        const result = await adapter.run({
          system: fixtures.system,
          prompt: testCase.prompt,
          tools: fixtures.tools,
          toolChoice: "auto",
          temperature: 0
        });

        await appendRecord({
          runId,
          startedAt,
          datasetVersion: config.datasetVersion,
          provider: adapter.provider,
          model: adapter.model,
          caseId: testCase.id,
          repeat,
          expected: testCase.expected,
          outcome: result.outcome,
          latencyMs: result.latencyMs,
          usage: result.usage,
          rawRequest: redact(result.rawRequest),
          rawResponse: redact(result.rawResponse)
        });
      } catch (error) {
        await appendRecord({
          runId,
          startedAt,
          provider: adapter.provider,
          model: adapter.model,
          caseId: testCase.id,
          repeat,
          harnessError: serializeError(error)
        });
      }
    }
  }
}

A temperature of zero can reduce variation, but it does not prove deterministic behavior. Providers may change serving infrastructure, model revisions, or decoding internals. Repeat each case enough times to observe instability rather than assuming it away.

Suggested project layout

function-benchmark/
├── fixtures/
│   ├── system.txt
│   ├── tools.core.json
│   ├── tools.compatibility.json
│   └── cases.jsonl
├── src/
│   ├── adapters/
│   │   ├── provider-a.ts
│   │   └── provider-b.ts
│   ├── normalize.ts
│   ├── validate.ts
│   ├── score.ts
│   └── run.ts
├── results/
├── package.json
├── tsconfig.json
└── benchmark.config.json

Minimal local commands

mkdir function-benchmark
cd function-benchmark
npm init -y
npm install ajv
npm install --save-dev typescript tsx @types/node
npx tsc --init
npx tsx src/run.ts --config benchmark.config.json

Install provider SDKs only for adapters you actually use, and pin their versions in the lockfile. Prefer environment-variable names scoped to each provider. Never write secrets into fixtures, result files, console snapshots, or version control.

{
  "datasetVersion": "1.0.0",
  "repeats": 5,
  "temperature": 0,
  "timeoutMs": 30000,
  "concurrency": 1,
  "models": [
    {
      "provider": "provider-a",
      "model": "replace-with-pinned-model-id"
    },
    {
      "provider": "provider-b",
      "model": "replace-with-pinned-model-id"
    }
  ]
}

Start with concurrency one. It makes rate limits and response ordering easier to diagnose. Increase concurrency only after the baseline is stable, then record it as part of the experiment.

Score selection and arguments separately

A single pass rate conceals important differences. A model can select the right tool but invent a required argument; another can produce perfect arguments for the wrong operation. Treat these as separate failures.

1. Request acceptance

Record whether the llm api accepted the request. Recommended categories are:

• accepted
• schema_rejected
• authentication_error
• rate_limited
• timeout
• provider_error
• harness_error

Do not count authentication, rate-limit, or harness failures as incorrect model decisions. Report them as coverage loss.

2. Tool selection

Build a confusion matrix with these outcomes:

• correct tool;
• wrong tool;
• tool called when abstention was expected;
• no tool when a call was expected;
• multiple calls when exactly one was expected;
• clarification when the fixture expected clarification;
• clarification when sufficient information was already present.

The last distinction matters. Asking a safe but unnecessary question may avoid a malformed call while still degrading the user experience.

3. Structural argument validity

Validate the untouched model arguments against the canonical schema. With Ajv:

import Ajv from "ajv";

const ajv = new Ajv({
  allErrors: true,
  strict: false,
  useDefaults: false,
  coerceTypes: false,
  removeAdditional: false
});

export function validateCall(
  schema: Record<string, unknown>,
  args: unknown
) {
  const validate = ajv.compile(schema);
  const valid = validate(args);

  return {
    valid: Boolean(valid),
    errors: validate.errors ?? []
  };
}

The disabled coercion, default insertion, and property removal are deliberate. They prevent the validator from improving the output before it is measured.

4. Semantic argument correctness

Schema validity is necessary but insufficient. weight_grams: 1 is a valid integer but is wrong when the user said “1 kg.” Compare values against the fixture after validation.

export function compareExpected(
  expected: Record<string, unknown>,
  actual: Record<string, unknown>
) {
  const missing = Object.keys(expected).filter((key) => !(key in actual));
  const wrong = Object.entries(expected)
    .filter(([key, value]) => key in actual && actual[key] !== value)
    .map(([key, value]) => ({
      key,
      expected: value,
      actual: actual[key]
    }));

  const unexpected = Object.keys(actual)
    .filter((key) => !(key in expected));

  return {
    exact: missing.length === 0 && wrong.length === 0 && unexpected.length === 0,
    missing,
    wrong,
    unexpected
  };
}

Some optional arguments admit multiple correct outputs. Encode explicit assertions rather than forcing exact object equality:

{
  "arguments": {
    "requiredEquals": {
      "email": "sam@example.test",
      "status": "shipped"
    },
    "optional": ["limit"],
    "constraints": {
      "limit": { "integer": true, "minimum": 1, "maximum": 20 }
    }
  }
}

5. Stability

For each model and case, group repeated normalized outcomes by a stable hash. A simple stability measure is:

stability = count(most_common_normalized_outcome) / successful_repeats

Report the numerator and denominator. A value without the number of successful repeats can be misleading, especially when provider errors removed observations.

Recommended report columns

Test validation and execution errors as conversations

Production llm function calling does not end when a call is emitted. The application may reject the arguments, the target system may reject the operation, or the result may reveal that the model needs another tool.

Add a second suite that executes deterministic fake tools. A mock tool should return predictable results and errors without contacting an external service.

export function fakeLookupOrder(args: unknown) {
  const validation = validateCall(lookupOrderSchema, args);

  if (!validation.valid) {
    return {
      ok: false,
      error: {
        code: "INVALID_ARGUMENTS",
        message: "The tool arguments did not match the schema.",
        details: validation.errors
      }
    };
  }

  const { order_id } = args as { order_id: string };

  if (order_id === "ORD-000000") {
    return {
      ok: false,
      error: {
        code: "ORDER_NOT_FOUND",
        message: "No order matched that order ID.",
        retryable: false
      }
    };
  }

  return {
    ok: true,
    order: {
      order_id,
      status: "shipped"
    }
  };
}

Use machine-readable error envelopes

{
  "ok": false,
  "error": {
    "code": "INVALID_ARGUMENTS",
    "message": "weight_grams must be an integer between 1 and 50000.",
    "field": "weight_grams",
    "retryable": true
  }
}

The model should not have to infer the error type from a stack trace. At the same time, do not expose internal exceptions, database details, or secrets in tool messages.

Recovery scenarios

1. Repairable type error: return that weight_grams must be an integer. Expect one corrected call.
2. Missing required value: return that postal_code is missing. Expect a user clarification, not an invented value.
3. Not found: return ORDER_NOT_FOUND. Expect an honest explanation or a request for another identifier, not repeated identical calls.
4. Transient failure: return a retryable error. Permit a bounded retry according to application policy.
5. Non-retryable failure: return a permission or policy error. Expect the model to stop.

Bound the conversation with maxToolRounds. A model that alternates between two invalid calls should not consume the entire request budget.

const MAX_TOOL_ROUNDS = 3;

for (let round = 0; round < MAX_TOOL_ROUNDS; round++) {
  const response = await adapter.continue(messages, tools);
  const calls = extractCalls(response);

  if (calls.length === 0) return finalize(response);

  const results = calls.map((call) => fakeToolRouter(call));
  messages.push(response.message);
  messages.push(...toProviderToolResults(calls, results));
}

return {
  kind: "stopped",
  reason: "max_tool_rounds_exceeded"
};

Score recovery precisely

A recovery pass should require the expected terminal behavior, not merely a second response. Record:

• number of tool rounds;
• whether the invalid call was repeated unchanged;
• whether the corrected arguments validate;
• whether a missing value was invented;
• whether the model stopped on a non-retryable error;
• whether the final user-facing statement agrees with the tool result.

Verify that the comparison is reproducible

Before interpreting model quality, verify the harness itself. The following checklist prevents many false conclusions.

1. Save a run manifest

{
  "runId": "2026-08-01T09-30-00Z",
  "datasetVersion": "1.0.0",
  "systemPromptSha256": "computed-at-runtime",
  "toolsSha256": "computed-at-runtime",
  "runnerCommit": "local-revision-or-archive-id",
  "runtime": {
    "node": "record-at-runtime",
    "platform": "record-at-runtime"
  },
  "settings": {
    "temperature": 0,
    "repeats": 5,
    "toolChoice": "auto",
    "timeoutMs": 30000,
    "concurrency": 1
  },
  "models": [
    {
      "provider": "provider-a",
      "requestedModel": "exact-model-id-used",
      "reportedModel": "capture-from-response-if-available"
    }
  ]
}

Compute hashes from the exact bytes sent by the runner. This catches accidental fixture edits and adapter transformations.

2. Snapshot adapter translations

For one representative case per provider, save a sanitized request fixture and test it locally. The snapshot should prove that:

• all three tools are present;
• required fields remain required;
• enums and numeric bounds survive translation;
• the system instruction is in the intended message field;
• the tool-choice mode is correct;
• no adapter has injected examples or provider-only hints.

3. Unit-test the scorer with deliberate mistakes

const scorerCases = [
  {
    name: "right tool and exact arguments",
    actual: call("lookup_order", { order_id: "ORD-104821" }),
    expectedScore: "pass"
  },
  {
    name: "right tool but wrong identifier",
    actual: call("lookup_order", { order_id: "ORD-104822" }),
    expectedScore: "semantic_failure"
  },
  {
    name: "wrong tool with plausible arguments",
    actual: call("search_orders", { email: "sam@example.test" }),
    expectedScore: "selection_failure"
  },
  {
    name: "unexpected extra property",
    actual: call("lookup_order", {
      order_id: "ORD-104821",
      email: "sam@example.test"
    }),
    expectedScore: "schema_failure"
  }
];

If the scorer cannot distinguish these cases, model comparisons will not be trustworthy.

4. Replay stored responses offline

Add a replay mode that reads sanitized raw responses, runs normalization again, and recomputes scores without contacting a provider. The replay should reproduce the original normalized records and summary.

npx tsx src/run.ts --config benchmark.config.json
npx tsx src/report.ts results/2026-08-01T09-30-00Z.jsonl
npx tsx src/replay.ts results/2026-08-01T09-30-00Z.jsonl
npx tsx src/report.ts results/replayed.jsonl

5. Compare hashes, not formatted output

Canonicalize object key order before hashing normalized outcomes. Do not reorder arrays unless their order is semantically irrelevant. Tool-call order may matter when one call depends on another.

6. Inspect failures manually

Metrics identify patterns; raw records explain them. Review at least one example from every failure category and every model. Check whether the problem belongs to the model, the provider protocol, your adapter, the validator, or an incorrectly specified fixture.

Failure cases worth testing explicitly

Wrong tool chosen from overlapping descriptions

Symptoms include using search_orders despite an exact order ID, or using lookup_order with an email address. Improve descriptions by stating both the positive condition and the boundary: “Use only when an exact public order ID is present.”

Do not immediately add the same rule to the system prompt, every tool description, and the user message. First change one variable and rerun the same cases. Otherwise you will not know which intervention helped.

Required arguments are guessed

A model may invent a postal code because the schema requires it. Include missing-information cases and score clarification explicitly. If tool use is forced, this test cannot measure natural clarification behavior.

Numbers and identifiers change type

Postal codes, account numbers, and telephone numbers should generally be strings. Quantities used in calculations should be numbers or integers. The 02139 case detects pipelines that accidentally turn a code into 2139.

Units are copied without conversion

If the schema expects grams and the prompt says kilograms, semantic validation must check the conversion. A structurally valid value of 1 is still wrong.

Unsupported enum synonyms

A model may emit "fast" when the schema allows only "express". Decide whether the model must map synonyms, the application may repair them, or the user must clarify. Measure the raw and repaired outcomes separately.

Extra arguments leak from the conversation

The model may carry an email address into lookup_order even though that tool does not accept it. Keep additionalProperties: false in the canonical schema and ensure the validator does not remove extras before scoring.

Malformed serialized arguments

Some protocols represent arguments as serialized JSON. Truncated output, invalid escaping, or surrounding prose must become a parse failure, not an empty object. Preserve the raw string for diagnosis.

Parallel calls appear unexpectedly

A model may request several functions in one response. Your adapter must retain every call and its identifier. For cases expecting one call, score additional calls as a policy violation even if the first call is correct.

Identical invalid calls loop

Hash the tool name and canonical arguments at every round. If the same failed call repeats after the model receives a non-transient error, stop early and classify the outcome as an unrecovered loop.

Text claims success before execution

A proposed call is not a completed action. Reject responses that tell the user an order was found, changed, or shipped before the application returns a successful tool result.

Provider schema rejection is mistaken for model failure

If one llm api rejects a keyword, the model never saw the request. Track schema compatibility independently and report the transformed schema used for any fallback run.

Retries bias the results

An SDK may automatically retry timeouts or rate limits. Record attempt counts when possible and apply the same retry policy across adapters. Model-level recovery after a tool error is different from transport-level retry.

Turn benchmark findings into production policy

The most accurate model in a small fixture is not automatically the best production choice. Use the report to build explicit runtime controls.

• Validate every call on the server, regardless of benchmark performance.
• Allowlist tool names instead of dynamically resolving arbitrary model text.
• Separate read-only tools from tools with side effects.
• Require confirmation or authorization for consequential operations.
• Set timeouts, retry limits, and maximum tool rounds.
• Use idempotency keys for retryable write operations.
• Log sanitized call decisions and validation outcomes.
• Keep a text fallback for provider outages or unsupported requests.

For high-risk tools, use a two-stage design: the model proposes a typed intent, then deterministic application logic checks permissions, state, and business rules before execution. Schema validity is not authorization.

A practical release gate

Define thresholds from your own risk tolerance and baseline data rather than copying universal numbers. A release gate can require:

• no regression on high-risk abstention cases;
• no invented values in missing-required tests;
• no unresolved validation errors in accepted calls;
• bounded recovery without repeated-call loops;
• a documented explanation for every schema transformation;
• successful replay of the stored evaluation records.

Keep raw counts beside percentages. “Nine of ten” and “nine hundred of one thousand” may display the same percentage while providing very different evidence.

Limitations of this benchmark

This method measures behavior on a controlled fixture, not universal model quality. Results are conditional on the exact prompts, schemas, provider adapters, settings, model versions, language, and date of execution.

• Hosted models and provider routing can change even when your application code does not.
• Temperature zero does not guarantee identical responses.
• Synthetic cases do not reproduce the full distribution of production conversations.
• The portable schema subset may underuse provider-specific capabilities.
• Exact-match scoring can penalize valid alternatives unless fixtures define acceptable variation.
• Latency and token usage can be recorded, but fair cost comparison requires the current terms and accounting rules of each provider.
• English-only fixtures do not establish multilingual reliability.
• Fake tools evaluate recovery logic but cannot reproduce every external-system failure.

Run two complementary suites: a portable suite for cross-provider comparison and a provider-optimized suite for the best production configuration on each platform. Label them clearly and never merge their scores.

Repeatable workflow

1. Write a canonical tool contract and freeze its version.
2. Create positive, confusable, abstention, argument, and clarification cases.
3. Implement thin adapters around each provider’s llm api.
4. Save sanitized raw requests and responses.
5. Normalize protocol syntax without repairing semantic errors.
6. Score request acceptance, tool choice, schema validity, and semantic accuracy separately.
7. Use deterministic fake tools to test recovery from structured errors.
8. Repeat cases and report stability with raw denominators.
9. Replay stored responses to verify the scorer and normalizer.
10. Promote findings into validation, authorization, retry, and release policies.

The important outcome is not a permanent leaderboard. It is a versioned, inspectable process that shows exactly where llm function calling succeeds or breaks in your application—and lets you rerun the same test after a model, prompt, schema, SDK, or provider changes.