≤

The passage around this formula

Whether automated LLM-judge evaluation becomes trusted for high-stakes decisions is not a third axis; it is mostly a readout of Axis A applied to one specific evaluator. A useful way to see the dependency is to write the automation decision as a threshold rule. Let $\hat\epsilon(a)$ be the estimated error rate of a judge on a decision class a , and let $\epsilon_{\max}(a)$ be the maximum error a policy is willing to tolerate for that stakes class. A defensible automation rule is $\text{Judge}(a) = \begin{cases} \text{automated}, & \hat\epsilon(a) \le \epsilon_{\max}(a) \ \text{and}\ \hat\epsilon(a)\ \text{externally audited} \\ \text{human required}, & \text{otherwise} \end{cases}$. The rule only licenses automation where both clauses hold, and the second clause is the one Axis A supplies or withholds. Zheng and colleagues’ own eighty-percent figure plausibly satisfies the first…

Read this part in the article →