$$A \succ B \iff s_A \ge s_B,\ \ c_A \le c_B,\ \ l_A \le l_B,\ \ \text{with at least one inequality strict.}$$

Equation 12 · Model Evaluation

Why Cost and Latency Belong in the Evaluation Score, Not a Footnote

This equation states a bound: one expression must stay on the indicated side of the other under the article’s assumptions.

Given that, the more defensible comparison between two agents is not a rank at all. It is a dominance relation. Say agent A Pareto-dominates agent B if A is at least as good on every axis and strictly better on at least one: $A \succ B \iff s_A \ge s_B,\ \ c_A \le c_B,\ \ l_A \le l_B,\ \ \text{with at least one inequality strict.}$. The Pareto frontier across a set of candidate agents is the subset that no other candidate dominates — the agents for which improving on any one axis would require giving something up on another. This relation makes no assumption about how much a point of accuracy is worth in dollars or seconds. It only says when one agent is unambiguously not worse than another. That is a weaker claim than a rank, and it is weaker on purpose: it is the claim the…

Meanings in this article

• $A$: at least as good on every axis and strictly better on at least one: [displayed formula].
• $s_A$: the success indicator or rate with subscript A (at least as good on every axis and strictly better on at least one: [displayed formula]).
• $l_A$: the expected latency with subscript A (at least as good on every axis and strictly better on at least one: [displayed formula]).

Equation guide → · Article →