multiplication

The passage around this formula

Peer-reviewed evidence, three different levers. DeepSeek-V2 replaces full multi-head keys and values with a low-rank latent projection — reducing $H_{\mathrm{kv}}$ $\cdot$ $d_h$ to a much smaller compressed dimension — and reports a 93.3 percent reduction in KV-cache size relative to the company’s own prior 67-billion-parameter dense model, while extending supported context to 128,000 tokens [ 15 ] . That shrinks the per-token cost of the equation above. StreamingLLM instead shrinks effective S : it keeps only a small window of recent tokens plus a handful of initial “attention sink” tokens, and reports enabling stable generation over sequences up to four million tokens with up to a 22.2-times…

Read this part in the article →