Equation 12 · Part 2 · How Benchmark Contamination Actually Works in Agentic Evaluation
Symbol k
What this part means
k is part of the quantity the equation computes from the expression on the right.
Its job in the formula
k is part of the quantity the equation computes from the expression on the right.
Full expression→Symbol k→Article meaning
The passage around this formula
…is designed to punish an agent whose competence is real but inconsistent across resampled trials — a stochastic policy with true per-trial success probability p has , which falls quickly as k grows. But a policy whose output on a given task is deterministic — an empty response, always, regardless of sampling — produces the identical transcript on every trial, so . A degenerate exploit is not merely invisible to pass@1; it is more invisible to pas, precisely…
Learn the underlying idea
A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.
Open the illustrated variables: a letter stands for a value guide →
See this notation across published equations →
Sources cited in the article section
- [8] Establishing Best Practices for Building Rigorous Agentic Benchmarks ↗
- [7] τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains ↗
- [9] Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation ↗
These citations provide research context; check each source for the exact claim it supports.