Equation 29 · Why Cost and Latency Belong in the Evaluation Score, Not a Footnote
What does this equation mean?
Read the formula alongside the article passage below. Each part has a deeper page with its role in the equation, the supporting passage and nearby citations.
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text. Read the equation part by part below; each part has a contextual explanation and a link to its mathematical background.
Read it piece by piece
Symbol epsilon
a small scheduling and aggregation overhead that grows slowly with k.
Symbol k
k is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
What the article says around this equation
where is a small scheduling and aggregation overhead that grows slowly with k . Two evaluations that both report “an agent retried up to five times” can therefore report latency figures that differ by nearly a factor of five, purely as an artifact of whether those five attempts ran one after another or side by side, with cost looking nearly identical between them. A latency number published without stating the retry and parallelism policy behind it is not comparable to another latency number published under a different policy, even when both describe the same underlying agent on the same task.
Sources cited in the article section
- [8] Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks ↗
- [10] Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores ↗
These citations give research context. Read each source to check which claims it supports.
Return to Why Cost and Latency Belong in the Evaluation Score, Not a Footnote