Measuring Tool Protocols and the Model Context Protocol: Evidence, Benchmarks, and Uncertainty

A protocol is not a capability

The Model Context Protocol has, in under two years, gone from an open-source release to something close to a default assumption: if a model is going to call a tool, there is a reasonable chance the connection is described in MCP’s terms. Anthropic’s original announcement framed the problem plainly — every new data source or tool required its own custom integration, an M×N problem that did not scale, and MCP was offered as the standardised M+N alternative, with day-one servers for Google Drive, Slack, GitHub, Git, Postgres, and Puppeteer ^[2]. That is a claim about integration cost, and it is a reasonable one. It is a different claim from “tool calling built on this protocol is reliable,” and the two get conflated constantly in adoption discourse.

This article stays on the second claim and asks what has actually been measured. Four questions recur across the literature: does the model call the right tool with the right arguments; how reliable is that call under repetition rather than as a single average; what does describing a large catalogue of tools cost in tokens and context budget; and how easily can a tool’s own description be used against the agent reading it. Each has an evidence base. None of the evidence bases agree with each other on a single number, and that disagreement is itself the most important finding in this area — more important than any individual benchmark score.

What the protocol actually specifies

Grounding the measurement question requires being precise about what MCP is and is not. Structurally, it is a JSON-RPC 2.0 message protocol connecting three roles: hosts, the LLM applications that initiate connections; clients, the connectors inside a host; and servers, the processes that expose capabilities. A server can offer resources (data for a model or user to read), prompts (templated workflows), and tools (functions the model may invoke); a client can, in the other direction, offer a server sampling, roots, and elicitation ^[1]. None of that architecture says anything about how reliably a given model selects among the tools it is offered, and the specification does not claim otherwise — its own security section is explicit that tool descriptions and annotations “should be considered untrusted, unless obtained from a trusted server,” and that hosts “must obtain explicit user consent before invoking any tool” ^[1]. That is a design requirement, not a measured property of any deployed system; whether real hosts actually enforce it, and how often models act on an untrusted description anyway, is precisely the kind of question this article is trying to separate from the specification’s own language.

It is worth being explicit about the difference in kind between the two paragraphs above. The specification’s transport and role structure is a fact — it is a document that says what it says. Its consent language is a normative requirement — a “must,” not a measurement of compliance. Anthropic’s framing of the integration problem MCP solves is a vendor claim about the protocol’s purpose, made by the party that designed it. None of these three is evidence about reliability, and the rest of this piece is concerned only with work that actually measured something.

Measuring whether the model calls the right tool

The most established benchmark for the underlying capability predates MCP itself. The Berkeley Function-Calling Leaderboard evaluates whether a model selects an appropriate function, formats the call correctly, fills arguments accurately, and — critically — abstains when no available function actually answers the request, using an abstract-syntax-tree comparison against annotated ground truth rather than executing every call for real ^[4]. Its authors report that current models perform strongly on short, single-turn invocations, but that “memory, dynamic decision-making, and long-horizon reasoning remain open challenges” once the evaluation moves into multi-turn, stateful, agentic settings ^[4]. That qualitative gap — solid on the easy slice, unreliable on the slice that resembles real deployment — recurs throughout this literature and is worth holding onto as a pattern rather than a single number.

A benchmark built specifically around live MCP servers rather than curated synthetic functions gives a sharper picture of where that gap actually sits. MCP-Bench connects twenty evaluated models to twenty-eight real MCP servers spanning two hundred and fifty tools across finance, travel, scientific computing, and academic search, and constructs tasks that require chaining tools together rather than invoking one in isolation ^[5]. The best model tested, GPT-5, reached a 100.0% valid-tool-name rate, 99.3% schema compliance, and 99.1% execution success — the call itself, mechanically, almost never breaks. Its overall task score was 0.749, driven down by a task-fulfillment component of 0.677 and a parallelism-efficiency component of only 0.339 ^[5]. A smaller model in the same study, Llama-3.1-8B-Instruct, kept comparably respectable schema compliance at 89.4% and a 90.9% execution-success rate, yet its tool-appropriateness score fell to 0.352 against GPT-5’s 0.767, and its parameter-accuracy score fell to 0.310 against GPT-5’s 0.749, driving its overall score down to 0.428 ^[5]. Put together, these numbers isolate exactly where the unreliability in tool calling actually lives: not in producing a syntactically valid call — that is close to solved even for a small model — but in choosing the right tool for an ambiguously phrased request and filling its arguments with values that are actually correct, which is where accuracy drops by a factor of two or more between models. This is a finding about these twenty models on these two hundred and fifty tools; it is not licence to rank vendors generally, since a different tool catalogue or task distribution can and does move these numbers substantially, a point the tool-count evidence below makes directly.

Reliability under repetition, not on average

A success rate reported as a single percentage hides a second question that matters more for anything actually put into production: if the same task is attempted again, does the same agent solve it again? τ-bench was built to answer exactly that question, simulating a language-model user interacting with a tool-using agent across retail and airline domains, each governed by written policy the agent is expected to follow ^[3]. Its authors introduce a metric they call pass^k — the probability that a fixed agent solves the same task instance successfully across all of k independent trials, distinct from the more familiar pass@k, which only requires one success in k attempts and therefore rewards best-case behaviour rather than dependable behaviour ^[3]. On this metric, a GPT-4o-based function-calling agent with an average task-success rate above 60% saw its pass^8 fall below 25% in the retail domain ^[3].

That gap is large enough to be worth writing down explicitly. If a task’s outcome were an independent Bernoulli draw with a fixed success probability $p$ equal to the reported average, the probability that all $k$ independent attempts at the same task succeed would be

At $p = 0.60$ and $k = 8$, that model predicts roughly 1.7%. The reported figure — under 25% — sits well above that naive prediction, and the direction of the gap is informative on its own: it is only possible if outcomes are not independent draws from one fixed probability, but rather reflect a task population that splits into instances the agent reliably solves and instances it reliably does not, with the reported 60% average blending the two. The practical consequence is that a single success-rate figure understates how often a system that “usually works” will keep failing on the same class of request every single time it is asked, and overstates how often a system that “usually fails” might still be coaxed into success by trying again. Averages compress that structure away; pass^k is one of the few published metrics designed to recover it, and its scarcity elsewhere in this literature is itself a gap in the evidence.

Why the numbers do not transfer between studies

None of the figures above are directly comparable to each other, and the reason is mechanical rather than a matter of which lab is more careful. BFCL scores structural correctness against curated, largely synthetic functions using an AST comparison that never executes a call ^[4]. τ-bench scores whether a simulated conversation ends in the correct database state, against two hand-built domains with a simulated user in the loop ^[3]. MCP-Bench scores execution against twenty-eight live third-party servers with their own quirks, rate limits, and undocumented edge cases ^[5]. Three different definitions of “the task succeeded,” three different populations of tools, three different degrees of exposure to the mess of a real running service. A model’s standing can, and does, move substantially between them, and no single number from any one of the three should be read as an estimate of “how reliable is tool calling” in general.

This is not a minor caveat. It is the central methodological fact governing everything else in this piece: a reported number is only ever a statement about a specific harness, a specific tool population, and a specific success criterion, on a specific date. Treat any claim that drops those qualifiers — “models call tools correctly 90% of the time” — as underspecified rather than false, and ask which of the three regimes above it was measured in before doing anything with it.

Measuring the cost of describing so many tools

A separate and more tractable question is what it costs, in tokens, to make a catalogue of tools available at all, since every tool’s schema and description are typically serialised into the model’s context before a single user turn is processed. A contributor to the MCP project’s own issue tracker measured this directly across a real tool suite: individual tool schemas ranged from roughly 103 to 1,024 tokens apiece, a tenfold spread within one suite, and a session registering twenty to thirty tools carried fifteen to thirty kilobytes of schema in context before any user message arrived — on the order of ten thousand tokens of pure overhead at typical per-tool sizes, with a measured production sample of 2,600 conversations over 22 days costing roughly $0.15 per conversation in first-turn schema tokens before prompt caching, and about $0.04 with a 75% cache-hit rate ^[10]. That is one measured deployment, not a universal constant, and the tenfold spread between the cheapest and most expensive tool schema in the same suite is itself evidence against treating “tokens per tool” as a fixed cost.

A simple linear model makes the assumption underneath that spread explicit. If a system prompt costs $T_0$ tokens before any tools are added, and each of $n$ tools costs on average $\bar{t}$ tokens to describe, total schema overhead is approximately

Under a fixed context budget $B$, the largest catalogue that fits is roughly $n_{\max} \approx (B - T_0)/\bar{t}$. The model is a reasonable first approximation, and it is also the assumption a tenfold per-tool variance directly undermines: $\bar t$ is not a constant of the tool, it is a property of how verbosely that tool’s schema and description happen to be written, which is an editorial choice rather than a physical fact about the function. A recent controlled study of exactly this trade-off tested fourteen models from 1.5B to 32B parameters plus one frontier API model across 6,566 controlled tool calls at three context budgets, and found that at an 8K-token budget uncompressed JSON schemas produced a near-zero 2.6% average exact-match accuracy, while a compression technique recovering 44 to 50% of the schema tokens lifted that same accuracy by an average of 20.5 percentage points across all eight models tested at that budget, with the gap narrowing to a point or less once the budget widened to 32K tokens ^[12]. The same paper estimated that frontier models could accommodate roughly 494 tools described as ordinary JSON schema before overflowing context, versus over 800 with compression applied ^[12] — a second confirmation that “how many tools can this system hold” is a joint function of budget and schema verbosity, not a fixed number attributable to the model alone.

Whether more tools also directly buys worse tool selection, independent of the token cost, is a related but separate question, and the evidence there is thinner and newer. A 2026 study frames it through what it calls a chance-corrected metric — comparing an agent’s tool-selection accuracy against what random selection would achieve at the same shortlist depth — and reports that on the BFCL tool population, an adaptive retrieval policy matched a full 370-tool listing’s 90.8% coverage using an average shortlist of only seven tools per query, while on the larger ToolBench population, a fixed shortlist of five tools reached 64.7% aggregate coverage and adaptive retrieval improved specifically on hard cases to 16.7% ^[11]. The headline claim worth taking from that study is narrower than it may sound: showing a model every available tool is not obviously better than showing it a well-chosen few, and past some catalogue size it can be worse, but the exact threshold at which that happens depends on the retrieval method and the tool population tested, not on a single universal number. Versions of a much more dramatic claim — specific accuracy collapsing from the low forties to single digits as a tool count grows past a few dozen — circulate widely in secondary commentary on this topic; where this article went looking for the primary source behind that particular figure, it could not find one that stated it, which is itself a small case study in how a plausible-sounding statistic outruns its own citation in a fast-moving area, and it is dropped here rather than repeated.

Measuring the attack surface

A tool description is not only an interface specification; in most current agent architectures it is also untrusted natural-language text the model reads and can be persuaded by, and both the specification’s own security section and the OWASP MCP Top 10 flag this directly ^{[1, 9]}. Invariant Labs coined the term tool poisoning attack for exactly this class of exploit: malicious instructions embedded in a tool’s description, invisible to the user reviewing the connection but fully visible to the model deciding whether to use it. Their April 2025 disclosure demonstrated a proof of concept against the Cursor coding agent in which a seemingly innocent add tool’s description instructed the model to read and exfiltrate the contents of SSH keys and other sensitive local files, and a second “shadowing” variant in which a malicious server, once co-present with a legitimate WhatsApp-integration server in the same session, could redirect messages the user had explicitly addressed elsewhere ^[8]. OWASP’s own entry for the category cites a related real-world case in which a compromised CI pipeline was used to remap an archive-management tool’s schema so that a benign-sounding operation triggered a destructive delete against production data ^[9].

Two later studies put a number on how often this class of attack actually succeeds rather than merely demonstrating that it can. MCPTox built 1,312 malicious test cases across ten risk categories and ran them against 45 live, real-world MCP servers offering 353 authentic tools, then measured how often twenty prominent agents fell for them; the highest measured attack-success rate was 72.8%, against o1-mini, and — the finding the paper’s authors emphasise most — greater general model capability did not track with greater resistance to this specific attack, with the best-performing model at refusing these attempts, Claude 3.7 Sonnet, still refusing fewer than 3% of them ^[6]. That is a result about this benchmark’s ten attack categories and this set of twenty agents; it says something specific and troubling about current safety training against this exploit class, not a general safety ranking across vendors, and should not be read as one. A separate, larger-scale census took the opposite approach — rather than crafting attacks, it scanned the existing ecosystem for tools already vulnerable to being weaponised, examining 12,230 real tools across 1,360 real servers and finding 1,062 tools (8.7%) and 370 servers (27.2%) exploitable through at least one attack pattern the authors tested for, without any user needing to interact with the malicious server directly ^[7]. Read together, these two results describe different halves of the same problem: a meaningful fraction of what is already deployed is exploitable in principle, and when a targeted attack is actually attempted, current models resist it only rarely and inconsistently.

Measuring latency and the uncertainty nobody reports

The weakest part of the public evidence base, by a wide margin, is anything to do with latency. Token overhead is measurable from a transcript after the fact, and several of the studies above did exactly that; end-to-end latency depends on serving infrastructure, network hops to third-party MCP servers, and retry behaviour under partial failure, none of which any of the benchmarks discussed here isolate or report with a stated measurement uncertainty. The production cost figures reported in the MCP project’s own issue tracker are the closest thing to a real-world operating measurement in this survey, and they describe cost, not latency, with no confidence interval attached to either the per-conversation dollar figures or the assumed cache-hit rate they depend on ^[10].

This matters because every number reviewed above is a point estimate standing in for a distribution, and almost none of the sources cited report the width of that distribution — a standard deviation across runs, a confidence interval on a success rate, or an explicit statement of how many trials a percentage was computed from. τ-bench’s pass^k metric is the one clear exception in this literature: it exists specifically because a single average success rate cannot distinguish a system that is uniformly somewhat unreliable from one that is bimodally very reliable on some tasks and unreliable on others ^[3]. Outside of that one metric, this survey found no widely cited tool-calling benchmark that routinely reports its own measurement uncertainty alongside its headline number, which is worth stating plainly as a gap rather than papering over: the field has more benchmarks than it has error bars.

Where the evidence is thin relative to the adoption

Collecting the sourced claims above against each other clarifies what kind of statement each one actually is. It is a fact, verifiable from the specification text itself, that MCP is a JSON-RPC protocol distinguishing tools, resources, and prompts, and that its own text requires user consent before tool invocation ^[1]. It is a vendor claim, attributable to Anthropic and worth treating as such, that the protocol meaningfully reduces integration cost relative to bespoke connectors ^[2]. It is measured evidence, with a stated benchmark and population, that mechanical call validity is close to solved while task-level tool appropriateness and parameter accuracy are not, and that reliability under repetition is markedly worse than average success rates suggest ^{[4, 5, 3]}. It is also measured evidence, from a different pair of studies with different methods, that a nontrivial fraction of already-deployed tools are exploitable and that targeted poisoning attacks succeed against current agents at rates that do not track general capability ^{[6, 7]}. And it is, at minimum, an open question rather than an established finding that any single “safe” tool-catalogue size exists independent of the retrieval method and schema verbosity in use ^{[11, 12]}.

What is conspicuously missing is any study that measures more than one of these axes on the same deployment at the same time — reliability, cost, and attack surface, together, on one real fleet of MCP servers under one accounting. Each axis currently has its own benchmark, its own tool population, and its own idea of what counts as success, which is precisely the fragmentation the earlier section on transferability described. Given that fragmentation, broad claims of the form “MCP makes agents reliable” or “tool calling is a solved problem” are neither confirmed nor refuted by the evidence surveyed here — they are simply larger than any single measurement anyone has actually taken.

Predictions, with the observations that would falsify them

These are forecasts, kept separate from the sourced findings above. Horizon: 22 August 2028.

One. A benchmark measuring reliability, schema-token cost, and susceptibility to tool-description attacks jointly, on one shared population of live MCP servers, will exist and gain citation traction. Disconfirmed if by 2028 the leading benchmarks in this area still each cover only one of the three axes.

Two. Adaptive or retrieval-based tool exposure will become the default integration pattern for any deployment offering more than roughly fifty tools, replacing the practice of registering an entire catalogue at session start. Disconfirmed if major MCP client implementations in 2028 still default to loading a server’s full tool list into every session regardless of catalogue size.

Three. Automated screening for tool-poisoning-style description attacks will become a standard pre-listing check for public MCP server registries, comparable to how package registries screen for known malware signatures today. Disconfirmed if the largest public MCP server directories in 2028 still perform no automated content screening of tool descriptions before listing.

Four. Reported success rates for what is nominally the same underlying capability will continue to diverge by twenty percentage points or more across benchmarks and harnesses, because the harnesses will keep measuring different things faster than anyone standardises them. Disconfirmed if two or more of the major current benchmarks in this space converge on a shared task population and success criterion, and their reported scores for the same models come within single digits of each other.

None of these four requires a capability breakthrough. They follow from the structure already visible in the evidence: fragmented benchmarks measuring different things, a token-cost model that degrades with tool-catalogue growth, and an attack surface that current safety training does not yet reliably close.

What to take away

Every strong claim made for tool-calling protocols in general, and MCP specifically, should be checked against a specific study, a specific harness, and a specific date before it is repeated. On the evidence actually collected here: mechanical call validity is close to solved, task-level tool selection and argument accuracy are not, reliability under repetition is substantially worse than a single average success rate implies, schema token cost scales with catalogue size in a way that current per-tool verbosity makes worse than it needs to be, and a meaningful share of already-deployed tools can be shown, empirically, to be exploitable by a description their user never reads. None of that is a case against the protocol. It is a case for treating “we adopted MCP” and “we know how reliable our tool calling is” as two entirely different sentences, and for asking, every time a number is quoted, which specific bench it was taken from.