What does this equation mean?

5. Non-independence between repeated trials. The confidence-interval formula above assumes each trial is independent of the others, and that assumption is often false in ways that specifically inflate confidence. Miller’s paper measures this directly: several widely used evaluation sets draw multiple questions from a shared context — several questions about the same passage, several sub-tasks from the same underlying scenario — and properly accounting for that clustering, rather than treating every question as its own independent draw, produced clustered standard errors “over 3x larger than naive standard errors” on the affected evals [ 8 ] . The general relationship is the one long used in survey statistics: if a naive standard error assumes independence, and trials instead arrive in clusters of average size m with intraclass correlation $\rho$ within a cluster, the correctly inflated standard error is $\mathrm{SE}_{\text{cluster}} = \mathrm{SE}_{\text{naive}} \times \sqrt{1 + (m - 1)\rho}$. Agent trials add a second, harder-to-audit source of the same problem: repeated attempts by one policy on one task instance are not independent draws either, because the same weights, the same training distribution, and often the same failure-prone reasoning pattern produce correlated outcomes across attempts. Shunyu Yao and colleagues built a reliability metric directly around this concern, measuring not just whether an agent succeeds at least once across several trials but whether it succeeds on every one of them; on their τ-bench retail domain, state-of-the-art function-calling agents that “succeed on <50% of the tasks” on a single trial saw that figure fall further, to below 25 percent, once the requirement was every one of eight trials succeeding rather than at least one [ 13 ] . Reporting the friendlier of those two numbers without the trial-independence caveat is a quiet but common way an accurate measurement becomes a misleading headline.