What does this equation mean?

Anthropic’s early red-teaming study gives the most granular public account of this labour market. The team recruited 324 crowdworkers, 307 through Mechanical Turk and 17 through Upwork, and collected 38,961 red-team attacks across model variants, with roughly 11,000 attacks logged against most model types [ 5 ] . Productivity was as concentrated as the labelling data above: about eighty percent of all attacks came from about fifty of the roughly three hundred workers. The paper’s central scaling finding is the one that matters most for this article’s argument — models trained with RLHF became significantly harder to red-team as they grew larger, while plain language models, prompted models, and rejection-sampling models showed a flat trend with scale [ 5 ] . Put in economic terms, if a single red-team attempt costs c and succeeds with probability p , the expected cost of finding one more genuine failure is $\mathbb{E}[C_{\mathrm{find}}] = \frac{c}{p}$. Holding c roughly fixed at crowdworker wage rates, a falling p as RLHF hardens a model means the expected cost per discovered failure rises mechanically with every round of safety training — the same defensive work that red-teaming is meant to validate makes the next round of red-teaming more expensive per useful finding, not less.