$$L_{\mathrm{total}} \approx L_{\mathrm{retrieve}} + k \cdot \ell_{\mathrm{rerank}} + L_{\mathrm{generate}}$$

Equation 11 · AI Agents & Systems

Building Production RAG: An Advanced Technical Guide

This equation gives an approximation: it relates the quantities while allowing an approximation.

The latency consequence follows from where the reranker sits in the request path. A simplified but useful model of end-to-end latency is $L_{\mathrm{total}} \approx L_{\mathrm{retrieve}} + k \cdot \ell_{\mathrm{rerank}} + L_{\mathrm{generate}}$. where k is the number of candidates handed to the reranker and $\ell_{\mathrm{rerank}}$ is the marginal latency per candidate scored. Recall generally improves as k grows, because a larger candidate set is more likely to contain the passage that actually answers the query — but k multiplies the one latency term a team most directly controls, so it is not a value to leave at a copied default. It is a dial to be set from a measured recall-versus- k curve against an explicit latency budget, the same way the hybrid weight $\alpha$ is set from a…

Meanings in this article

• $k$: the number of candidates handed to the reranker.

Equation guide → · Article →