What does this equation mean?

The tau-bench numbers make the distinction concrete rather than abstract. Testing function-calling agents built on frontier models against realistic retail and airline customer-service scenarios — each requiring the agent to follow domain policy, call the right APIs, and leave the backend database in a state matching an annotated goal — the authors found that even strong agents succeeded on well under half of tasks at pass@1, and that reliability across repeats fell sharply as k increased: an agent’s pass ^k score in the retail domain dropped below 25% by k=8 , and airline-domain performance, already lower at k=1 , degraded further from there [ 3 ] . Put in words rather than symbols: an agent that a casual observer would call “usually right” on any given try is, on the paper’s own numbers, more likely than not to slip up at least once if you ask it to repeat the same class of task eight times. The authors’ own conclusion is not that the agents were unusually bad — several were the strongest publicly available systems at the time — but that consistency, not raw single-shot competence, was the thing missing [ 3 ] .