What does this equation mean?

The size of the win has a clean shape. Model the draft’s acceptance probability as $\alpha$ per token, roughly constant and independent across the $\gamma$ tokens proposed in a round — an idealization real traffic does not fully satisfy, but a useful one for seeing the ceiling. The expected number of tokens accepted per verification round is then $\mathbb{E}[\text{tokens per round}] = \frac{1-\alpha^{\gamma+1}}{1-\alpha}$. This rises with both $\alpha$ and $\gamma$ , but with steeply diminishing returns in $\gamma$ for any $\alpha$ below one: drafting fifty tokens ahead does not buy anywhere near fifty accepted tokens, because the marginal proposal deep into a long draft is unlikely to be exactly what the target would have generated. The bottleneck decode started with reappears one level down — the draft model is itself a small, serial, memory-bound process, so speculative decoding is not a free lunch; it trades some of decode’s own bandwidth cost for a second model running continuously, plus wasted target-model computation on every rejected proposal. Its win is largest where the roofline argument above says spare compute is most available: at low batch sizes, where the target model is memory-bound and its arithmetic units sit mostly idle during an ordinary decode step anyway. That idle arithmetic verifies the extra proposed tokens at close to no additional cost.