$$B_{\text{day}} = k_1 n_1 c_1 + k_2 n_2 c_2$$

Equation 15 · Model Evaluation

Building a Custom Evaluation Suite for a Production Agent

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

The practical answer most teams converge on is a tiered budget, and it is worth writing the tiering down as an explicit equation rather than an implicit habit, because an implicit habit is what quietly erodes into “we stopped running the full suite” a few months in. If every merge triggers a cheap smoke tier of $n_1$ tasks at unit cost $c_1$ , run $k_1$ times a day, and a slower, more thorough tier of $n_2$ tasks at unit cost $c_2$ , run only $k_2$ times a day, the daily evaluation bill is simply $B_{\text{day}} = k_1 n_1 c_1 + k_2 n_2 c_2$. The first term is the per-merge tax — it has to stay small enough that engineers do not start skipping it, which is the single most common way a CI gate quietly stops functioning. The…

Equation guide → · Article →