Why Cost and Latency Belong in the Evaluation Score, Not a Footnote

Picture two candidate support agents up for the same deployment decision. Agent A resolves 95 percent of incoming tickets correctly, at an average cost of two dollars and forty cents per ticket and a median response time of forty seconds. Agent B resolves 90 percent, at thirty-one cents per ticket and six seconds. Both figures are illustrative rather than measured — the point of the example is structural, not empirical. Which agent is better?

There is no answer until someone states what the deployment can tolerate. A high-value legal-review queue handling five hundred tickets a month might happily pay the premium for the extra five points of accuracy. A consumer chat surface fielding five million tickets a month, where a forty-second wait loses the customer before the answer arrives, would be ruined by Agent A regardless of its accuracy. The two agents are not ranked against each other by any fact about the agents themselves. They are ranked by a decision the evaluator has to make about how much accuracy is worth, in dollars and in seconds, for this particular use.

Most published agent evaluation still proceeds as though that decision does not exist. A benchmark reports a resolution rate, a pass rate, an exact-match score, and orders the field by that single number, with cost and latency mentioned in passing if at all — an appendix table, a caveat about the hardware used, a number quietly dropped between benchmark revisions. This article argues that treating cost and latency as anything other than first-class, scored dimensions of an agent evaluation is a methodological error, not a simplification. It looks at how a small but growing set of independent evaluators have started to correct that error, and at how easily two agents’ cost and latency numbers can be made to look comparable when the conditions under which they were actually measured were not.

A leaderboard rank compresses three numbers into one

Treat a single evaluation run as producing a summary triple: a success indicator or rate $s$, an expected cost $c$, and an expected latency $l$. A leaderboard rank is a total order — every agent gets exactly one position — built by applying some function to that triple, almost always a function that looks only at $s$ and discards $c$ and $l$ entirely, or folds them into a secondary table nobody reads before making a decision.

The trouble with discarding $c$ and $l$ is not that they are minor details. It is that “cost” and “latency” are themselves not single, unambiguous numbers waiting to be read off a meter. Dehghani and colleagues made this point rigorously for model efficiency in general, cataloguing the many different cost indicators researchers use — parameter count, FLOPs, throughput, peak memory, wall-clock time — and showing experimentally that these indicators routinely disagree with one another: an architecture that looks efficient on one indicator looks inefficient on another, and reporting only one or two of them, as is common practice, produces a partial and sometimes actively misleading picture of what a model actually costs to run ^[6]. An agent adds a further layer on top of that problem, because its cost and latency are not fixed properties of a checkpoint the way parameter count is; they are properties of a whole trajectory of tool calls, retries, and context growth, measured under whatever conditions the evaluator happened to run.

Given that, the more defensible comparison between two agents is not a rank at all. It is a dominance relation. Say agent $A$ Pareto-dominates agent $B$ if $A$ is at least as good on every axis and strictly better on at least one:

The Pareto frontier across a set of candidate agents is the subset that no other candidate dominates — the agents for which improving on any one axis would require giving something up on another. This relation makes no assumption about how much a point of accuracy is worth in dollars or seconds. It only says when one agent is unambiguously not worse than another. That is a weaker claim than a rank, and it is weaker on purpose: it is the claim the data alone actually supports, before anyone’s judgment about relative value has been added to it.

The multi-objective evaluation problem, formally

Dominance alone will not settle every comparison, because most pairs of agents are not comparable in this weak sense — one is more accurate, the other is cheaper or faster, and neither dominates. Agent A and Agent B from the opening example are exactly such a pair: A wins on $s$, B wins on both $c$ and $l$, and the Pareto relation is silent about which is “better.” Something has to break the tie, and that something is necessarily a value judgment about how many dollars, or how many seconds, one point of success is worth to this particular deployment.

Make that judgment explicit and it becomes a scalarized score. Choose exchange rates $\lambda_c$ (accuracy points per dollar) and $\lambda_l$ (accuracy points per second of latency), and define

This is a legitimate way to rank agents, and it is what most procurement decisions ultimately have to do. But writing it out this explicitly makes the hidden step visible: the ranking that comes out depends entirely on $\lambda_c$ and $\lambda_l$, numbers that are not properties of the agents at all. A call center with razor-thin per-ticket margins and a research lab doing one-off due-diligence work on a handful of contracts will set $\lambda_c$ at wildly different values, and the two will reasonably reach opposite conclusions about which agent to deploy, from identical measured data. A leaderboard that publishes a single blended score has made this choice for every reader, usually without saying so, and usually without saying what the choice was.

This is exactly why the more careful independent evaluators have converged on publishing the raw triple rather than a single blended figure. Vals AI, an independent third-party evaluator that builds benchmarks for professional domains such as law, tax, and finance, states its position on this directly: “benchmarks often report only accuracy numbers; however, it is important to consider factors such as efficiency, cost, time taken per test, failure modes, and more” ^[4]. Its published results report accuracy, latency, and cost as three separate figures per model per benchmark, alongside error bars meant to reflect statistical uncertainty in each one ^[4] — leaving the weighting between them to the reader rather than baking a $\lambda_c$ and $\lambda_l$ into the ranking itself. That is a defensible editorial choice, and it is a minority one; most agent benchmarks still publish only $s$.

How independent evaluators already do this

The clearest working example of Pareto-frontier reporting applied to language models comes from Artificial Analysis, an independent benchmarking outlet that publishes a composite Intelligence Index alongside pricing and speed data gathered across API providers. Its published methodology is explicit about standardizing the measurement rather than leaving it to whatever a vendor chooses to disclose: token-per-second figures are normalized to a common tokenizer so that models with different tokenization schemes remain comparable, and a blended price is computed from actual token usage across the evaluation workload rather than from sticker price per token, because, as the methodology states, “models that produce longer answers or more reasoning tokens will have a higher cost per task, even at identical per-token prices” ^[1]. That single sentence is worth pausing on: it says that cost, properly measured for an agent or a reasoning model, is an emergent property of a trajectory’s length, not a lookup from a pricing table, which is precisely the kind of context-dependence a footnote cannot capture.

An independent analysis of Artificial Analysis’s published pricing and intelligence data, conducted by Digital Applied, applied a strict Pareto-dominance filter across all three axes — quality, blended price, and output speed — to the frontier and near-frontier models it tracked in the second quarter of 2026, and reported that only six of the twenty models examined survived as non-dominated; the other fourteen were each beaten on every one of the three axes simultaneously by some cheaper or faster alternative ^[2]. That finding belongs to that specific analysis of that specific public dataset at that date, not to Artificial Analysis’s own official ranking, and it should be read as an illustration of what a dominance filter does to a crowded field rather than as a verdict on any named model: a majority of entrants in a competitive market, once cost and speed are checked alongside quality, turn out to add nothing that a cheaper or faster rival does not already offer.

Set this against a leading benchmark that does the opposite. Epoch AI’s SWE-bench Verified evaluation is, on reproducibility grounds, one of the more careful agent benchmarks in active use: it documents that it evaluates 484 of the 500 available instances, excluding sixteen that do not run reliably in its own infrastructure and stating the specific reason for each exclusion, and it versions its evaluation harness explicitly, noting that a February 2026 upgrade to the harness changed reported scores enough that its default results view only shows scores from the new version onward ^[3]. That is exactly the kind of methodological discipline this article is arguing for — applied entirely to the accuracy axis. Nowhere in that same published evaluation is a cost-per-instance or cost-to-solve figure reported alongside the resolution rate ^[3]. A benchmark can be rigorous about reproducibility and still silent about cost; the two disciplines have not yet arrived together, even at the more careful end of the field.

Cost as a first-class score, not a side reading

The sharpest form of the argument is a benchmark that puts cost inside the pass/fail criterion itself, rather than reporting it beside a separately computed success rate. CostBench does this directly: it evaluates agents on multi-turn tool-use planning tasks, set in a travel-booking domain, where multiple valid tool sequences solve the same task at different total costs, and where the environment introduces dynamic disruptions — tool failures, price changes — mid-task, so that an agent has to replan around a shifting cost landscape rather than execute a fixed plan ^[9]. Its authors report that even leading models achieve under 75 percent exact-match on the hardest task tier, and that performance degrades by roughly 40 percent once dynamic disruptions are introduced ^[9]. What makes this a genuinely different kind of measurement is that the metric is not “did the agent complete the task” scored separately from “how much did it cost” — an agent that completes the task via an expensively wasteful tool sequence is simply marked wrong, the same way an agent that fails outright is marked wrong. Cost is inside the definition of success rather than appended to it.

A softer but more widely deployed version of the same idea appears in production LLM cascades. FrugalGPT treats the choice among available LLM APIs, which differ in price by as much as two orders of magnitude, as an optimization problem to be solved per query rather than a single global choice of “the best model”: a cheap model is tried first, and a query is only escalated to a more expensive one when the cheap model’s own confidence signals warrant it. Chen and colleagues report that this cascading strategy can match the accuracy of using the strongest model on every query while cutting cost by up to 98 percent, or, held to the same total cost, improve on that model’s accuracy by around 4 percent ^[5]. Those are the paper’s own reported figures for its own method and should be read as such — a research result on the datasets it was tested against, not a guarantee that transfers to every workload. What the result demonstrates structurally, independent of the specific numbers, is that a cascade is an algorithm for walking the accuracy-cost Pareto frontier at inference time, one query at a time. That algorithm only makes sense to build at all once cost has been accepted as a first-class axis worth optimizing against, rather than a number to be reported after the fact.

What a mature discipline of latency measurement looks like

Agent evaluation is not the first field to face this problem, and it is worth looking at one that solved it earlier. Hardware and inference-serving benchmarking faced the identical difficulty — a “how fast” number is meaningless without specifying exactly what workload pattern produced it — and MLPerf’s Inference suite is the industry’s answer. It defines a fixed set of measurement scenarios rather than a single ad hoc timing: single-stream, multi-stream, server, and offline, each simulating a different realistic pattern of how requests actually arrive, evaluated by “a standard load generator generating inference requests in a particular pattern and measuring a specific metric” that is fixed in advance rather than chosen after the fact by whoever is running the benchmark ^[7]. Its published results report percentile latency figures — the shape of the latency distribution, not just its mean — precisely because a mean response time hides the tail that a real user actually experiences ^[7].

Nothing about MLPerf’s specific scenarios transfers directly to agent evaluation; an agent trajectory is not a single inference request, and an agent benchmark’s realistic arrival patterns look nothing like a datacenter’s. But the discipline underneath does transfer directly: name the measurement scenario before reporting the number, fix it in advance rather than choosing it after seeing which one flatters the result, and report the distribution rather than a single point estimate. Agent evaluation, at its current state of practice, mostly does none of the three.

The confound: temperature, retries, and parallelism

Even granting that an evaluator wants to report cost and latency honestly, doing so comparably across two different agents turns out to be genuinely hard, because both quantities are sensitive to measurement choices that rarely get disclosed alongside the headline number.

Start with how an agent handles its own failures. Suppose a task is retried up to $k$ times under two different execution policies: sequential retry, where each attempt waits for the previous one to finish before starting, and parallel sampling, where all $k$ attempts run concurrently and the first success is taken. Cost is roughly indifferent to which policy was used — the compute consumed scales with the number of attempts made, $C(k) \approx k\bar{c}$, regardless of whether they ran one after another or all at once. Latency is not indifferent at all:

where $\epsilon(k)$ is a small scheduling and aggregation overhead that grows slowly with $k$. Two evaluations that both report “an agent retried up to five times” can therefore report latency figures that differ by nearly a factor of five, purely as an artifact of whether those five attempts ran one after another or side by side, with cost looking nearly identical between them. A latency number published without stating the retry and parallelism policy behind it is not comparable to another latency number published under a different policy, even when both describe the same underlying agent on the same task.

The problem runs deeper than policy disclosure, because the measurement apparatus itself can distort the number it reports. Chandrasekar and Kramberger identify a specific and previously undocumented source of this distortion: widely used LLM inference benchmarking tools are built on single-process, asyncio-driven client architectures, and under Python’s Global Interpreter Lock, that architecture creates a client-side queuing bottleneck that inflates the reported time-to-first-token and time-per-output-token as the number of concurrent requests rises — a bias in the measuring instrument, not in the system being measured, that grows specifically at the higher concurrency levels many evaluators use to simulate realistic parallel-agent load ^[8]. Their proposed fix is a multi-process evaluation architecture that removes the client-side bottleneck, plus a normalized latency metric designed to remain comparable across differing sequence lengths ^[8]. The finding is a caution specific to that class of benchmarking tool, not a claim that every published latency figure in the field is wrong, but it establishes concretely that “we measured latency” is not by itself a sufficient methods statement — the tool doing the measuring has its own performance characteristics that can leak into the result.

Temperature adds a third source of incomparability, this time on the accuracy side rather than the cost or latency side, though it has direct consequences for how much repeated measurement any of this requires. Blackwell and colleagues document that large language models remain stochastic in their outputs “even when setting temperature to zero with a fixed random seed” ^[10], meaning a single run at a fixed sampling configuration does not fully pin down what a model will do next time. Their proposed remedy is to quantify a benchmark score’s uncertainty through repeated trials rather than trusting a single run, and they note explicitly that few benchmark studies do this, in part because of the added experimental cost of running enough trials to make the uncertainty estimate meaningful ^[10]. That last point closes a loop this article has been building toward: getting an honest, repeated-trial estimate of an agent’s cost and latency, with real confidence intervals rather than a single anecdotal run, itself costs money and time — a second-order cost that is essentially never disclosed, and that shrinks in direct proportion to how confident anyone can be that the first-order cost and latency numbers reported are the ones a deployment would actually see.

A minimal reporting discipline

None of the fixes implied by the sections above require new infrastructure, only a change in what gets published alongside a headline score.

Report the triple, not the rank. Publish $(s, c, l)$ per agent per task suite, and let a scalarized ranking, if one is needed for a specific procurement decision, be computed downstream with weights that are stated rather than assumed.

Disclose the measurement conditions with the same care given to the task suite itself: sampling temperature, the retry and parallelism policy in force, the number of independent trials behind any averaged figure, the hardware or API provider used, and the date, since provider-side pricing and serving infrastructure both change over time.

Prefer a Pareto frontier over a single blended score whenever the evaluation is meant to inform more than one downstream use case, because the frontier makes no assumption about how the reader values a dollar against a point of accuracy, while a blended score always does, whether or not that assumption is stated.

When a scalarized score is unavoidable — a single procurement decision, for a single stated deployment — publish $\lambda_c$ and $\lambda_l$ explicitly next to the resulting number, so a reader with a different cost tolerance can substitute their own weights rather than inherit the evaluator’s.

Predictions, with the observations that would falsify them

These are forecasts, separated deliberately from the sourced material above. Horizon: 12 August 2028.

One. Independent leaderboards for agentic systems will increasingly default to a Pareto-frontier or explicit three-axis display — quality, cost, and latency — rather than a single ranked list, following the pattern already visible in cost-and-quality benchmarking for base models. Disconfirmed if the most-cited agent leaderboards in 2028 still present a single accuracy-ranked list as their primary view, with cost and latency, if present at all, confined to a secondary table.

Two. Disclosure of measurement conditions — trial count, sampling temperature, retry and parallelism policy, and evaluation date — will become an expected methods-section item for agent benchmark papers, comparable to how dataset composition is already expected. Disconfirmed if leading agent benchmark publications in 2028 continue to report point cost and latency estimates with no disclosure of how many trials, and under what concurrency regime, produced them.

Three. Benchmarks that fold cost directly into the pass/fail criterion, in the manner CostBench does, will move beyond research proposals into at least one widely cited procurement or leaderboard context. Disconfirmed if, by 2028, cost-integrated scoring of this kind remains confined to academic benchmark papers with no adoption in an evaluation used for actual purchasing decisions.

None of these requires a capability discontinuity in the underlying models. They follow from a field noticing what independent evaluators such as Artificial Analysis and Vals AI have already started to demonstrate: that a success rate reported alone is an incomplete description of what was measured, and that the missing pieces are not hard to publish once an evaluator decides they are worth publishing.

What to take away

An agent evaluation that reports only a success rate has made two decisions silently: that cost and latency do not matter enough to score, and, if they are mentioned at all, that some unstated exchange rate between accuracy and resource use is the right one for every reader. Neither decision is a fact about the agent being evaluated. Both are value judgments smuggled into what looks like a measurement.

The repair is not exotic. Score the triple. State the dominance relation before reaching for a rank. If a single number is genuinely required, publish the weights that produced it. Disclose the temperature, the retry policy, and the parallelism behind every cost and latency figure, because none of those numbers means anything portable without them. A handful of independent evaluators have already started doing this — weighing cost with the same care given to accuracy, timing latency rather than assuming it away, filtering candidates by dominance before ranking what is left. The rest of the field has the harder problem solved for it; what remains is deciding that cost and latency were never footnotes to begin with.