What does this equation mean?

Finding the failure is only half the design problem. The other half is scoring it, and a plain pass/fail count throws away exactly the information a consequence-aware evaluation needs — it treats a wrong word choice and a deleted database as the same unit. A more defensible score weights each discovered failure mode by its consequence rather than counting it once: $R = \sum_{i} \Pr(\text{failure mode } i) \cdot B_i$. where $B_i$ is the blast radius assigned to failure mode i : how much of a system, how much data, how many users or how much liability a given failure could plausibly reach, drawn from a small number of severity tiers rather than treated as a continuous unknown. This is not a hypothetical scoring scheme; tiered, severity-weighted evaluation of exactly this shape is already how the field’s two most cited frontier-risk frameworks operate at the level of an entire model. Anthropic’s Responsible Scaling Policy defines a ladder of AI Safety Levels, modelled loosely on biosafety-level containment standards, under which a model’s required safety, security and deployment safeguards scale with the severity of catastrophic risk its own capability evaluations place it at [ 11 ] . OpenAI’s Preparedness Framework runs a parallel structure at the level of specific risk domains — cybersecurity, chemical/biological/radiological/nuclear capability, persuasion, and model autonomy — scoring each as Low, Medium, High or Critical, and committing not to deploy a model that scores High in a given category until mitigations bring the score back down to Medium [ 12 ] . Neither framework was built for scoring a single agent’s task-level failures; both were built for gating whether an entire model ships. The analogy worth drawing is not that the two problems are identical, but that the field’s own most consequential risk-management structures already reject an averaged, ungraded pass/fail count in favour of exactly this kind of tiered, severity-weighted scoring — and an evaluation harness built for consequential agent tasks has good reason to borrow the same shape at a smaller scale, alongside the general risk-taxonomy work national standards bodies have separately published for generative systems more broadly [ 9 ] .