Inference Economics
How AI Inference Serving Actually Works
Every request is answered by two machines sharing one name: a compute-bound pass across the prompt, then a memory-bound crawl through every token after it. Batching, caching, speculation, quantization and expert routing all exist to manage that one seam.