≈

The passage around this formula

The latency consequence follows from where the reranker sits in the request path. A simplified but useful model of end-to-end latency is $L_{\mathrm{total}} \approx L_{\mathrm{retrieve}} + k \cdot \ell_{\mathrm{rerank}} + L_{\mathrm{generate}}$. where k is the number of candidates handed to the reranker and $\ell_{\mathrm{rerank}}$ is the marginal latency per candidate scored. Recall generally improves as k grows, because a larger candidate set is more likely to contain the passage that actually answers the query — but k multiplies the one latency term a team most directly controls, so it is not a value to leave at a copied default. It is a dial to be set from a measured recall-versus- k curve against an explicit latency budget, the same way the hybrid weight $\alpha$ is set from a…

Read this part in the article →