≈

The passage around this formula

Start with how an agent handles its own failures. Suppose a task is retried up to k times under two different execution policies: sequential retry, where each attempt waits for the previous one to finish before starting, and parallel sampling, where all k attempts run concurrently and the first success is taken. Cost is roughly indifferent to which policy was used — the compute consumed scales with the number of attempts made, C(k) $\approx$ k$\bar{c}$ , regardless of whether they ran one after another or all at once. Latency is not indifferent at all: $L_{\text{sequential}}(k) \approx k\,\bar{l}, \qquad L_{\text{parallel}}(k) \approx \bar{l} + \epsilon(k)$. where $\epsilon(k)$ is a small scheduling and aggregation overhead that grows slowly with k . Two evaluations that both report…

Read this part in the article →