$$L_{\text{sequential}}(k) \approx k\,\bar{l}, \qquad L_{\text{parallel}}(k) \approx \bar{l} + \epsilon(k)$$

Equation 28 · Model Evaluation

Why Cost and Latency Belong in the Evaluation Score, Not a Footnote

This equation gives an approximation: it relates the quantities while allowing an approximation.

Start with how an agent handles its own failures. Suppose a task is retried up to k times under two different execution policies: sequential retry, where each attempt waits for the previous one to finish before starting, and parallel sampling, where all k attempts run concurrently and the first success is taken. Cost is roughly indifferent to which policy was used — the compute consumed scales with the number of attempts made, C(k) $\approx$ k$\bar{c}$ , regardless of whether they ran one after another or all at once. Latency is not indifferent at all: $L_{\text{sequential}}(k) \approx k\,\bar{l}, \qquad L_{\text{parallel}}(k) \approx \bar{l} + \epsilon(k)$. where $\epsilon(k)$ is a small scheduling and aggregation overhead that grows slowly with k . Two evaluations that both report…

Meanings in this article

• $\epsilon$: a small scheduling and aggregation overhead that grows slowly with k.

Equation guide → · Article →