subscript

The passage around this formula

An aggregate benchmark score is a weighted average over items or tasks: $S = \sum_{i=1}^{n} w_i \, a_i$. where $a_i$ is accuracy, or another per-task quality measure, on task i , and $w_i$ is that task’s share of the total, with the weights summing to one. The arithmetic makes the trap explicit: S can be almost unchanged by compression even when some single $a_i$ collapses, provided $w_i$ for that task is small relative to the rest, or provided a rise elsewhere offsets the fall. A single published number, ninety-eight point six percent of baseline, or a small perplexity increase, cannot by construction distinguish “compression cost nothing” from “compression cost a great deal on one task nobody weighted…

Read this part in the article →