=

The passage around this formula

and contrast it with a severity-weighted version, $R = 1 - \sum_{i=1}^{n} w_i \cdot \mathbb{1}[\text{trial } i \text{ failed}]$. where $w_i$ scales each failure by how costly it actually was. Two agents can share an identical $\hat{p}$ of, say, ninety-five percent, while one of them fails harmlessly - an unhelpful but reversible answer - and the other fails catastrophically - an irreversible transaction, a deleted repository, a wrong medical dosage recommendation - on that same five percent. No standard agent benchmark publishes R , because assigning a defensible $w_i$ requires a judgment about real-world consequence that a replayable, sandboxed task suite is not built to carry, and because the tasks that would carry the highest weights are, not…

Read this part in the article →