Symbol barc_inf

The passage around this formula

Which combination a provider chooses is not primarily a research question. Write total lifetime cost as $C_{\mathrm{total}} = C_{\mathrm{train}} + Q \cdot \bar{c}_{\mathrm{inf}}$. with Q requests over the model’s life. When Q is small — a research artefact, an internal tool — the training term dominates and compute-optimal allocation is the right target. When Q is very large, the second term dominates by orders of magnitude, and every technique that reduces $\bar{c}_{\mathrm{inf}}$ pays back continuously while every technique that reduces $C_{\mathrm{train}}$ pays back once.

Read this part in the article →