Building a Custom Evaluation Suite for a Production Agent

The public benchmark is not your suite

A team shipping a production agent almost always has SWE-bench, WebArena, GAIA, or tau-bench numbers on hand somewhere — a model card, a vendor blog post, a spreadsheet from evaluation season. None of those numbers describe the system actually running in front of customers. They describe a different agent, wired to different tools, tested against a different task distribution, graded by a different harness. A custom production system needs a custom suite, and building one well is largely not a modeling problem. It is an organizational one: which cases go in, who decides what a pass means, how the checks run on every change without becoming a bottleneck, how much the checking is allowed to cost, and how the whole thing is kept from rotting once it stops being new.

Anthropic’s own engineering guidance on building agent evals states the starting point plainly: begin with the manual checks already run before each release and the tasks real users already attempt, and if the system is already live, look at the bug tracker and the support queue ^[1]. That is a small, almost unglamorous instruction, and it is the correct one. It says the first evaluation dataset is not designed, it is recovered — pulled out of records the team already has, rather than imagined from a whiteboard. Everything that follows in this piece is an elaboration of that one sentence: how to recover the cases properly, how to grade them defensibly, how to run the checks continuously without drowning the team in compute, and how to stop the resulting suite from becoming exactly the kind of stale, gameable artifact it was built to prevent.

Deriving representative cases from what actually happened

The instinct when a team decides it needs an eval suite is to write test cases the way a textbook would: enumerate the agent’s intended capabilities, then invent a scenario for each one. This produces a suite that measures whether the agent can do what its designers already believed it could do, which is close to measuring nothing. The corrective is to derive cases from the system’s own operating history rather than from a designer’s mental model of the system.

Anthropic’s recommendation is concrete: twenty to fifty simple tasks drawn from real failures is a credible starting point, not a placeholder to be embarrassed about, because early in a suite’s life each change tends to have a large, clearly visible effect, so a small sample is still informative ^[1]. Hamel Husain, describing the same recovery process from the consulting side, is more insistent about the reading itself: examine the traces and failure modes critically, and when starting out, read essentially everything — all the traces generated by test cases and by real users — rather than sampling from the outset, because it is impossible to know in advance which five percent of the log actually contains the interesting failure ^[2]. His practical stopping rule is almost anti-methodological and better for it: keep reading logs until the sense of learning something new from each additional one stops. That is not a sample size, it is a saturation signal, and it works because early logs are cheap relative to what they reveal.

This process has an old name that predates language models by decades. Google’s site reliability engineering practice treats a postmortem as a written record of an incident, its impact, the actions taken to resolve it, the root cause, and specifically the follow-up actions meant to prevent the incident recurring, triggered by concrete criteria such as user-visible degradation past a threshold, any data loss, or an on-call engineer having to intervene manually ^[12]. An eval case pulled from an incident log is functionally a postmortem action item that has been converted into a regression test instead of a paragraph of prose. The SRE literature also recommends aggregating postmortems across teams specifically to find common themes rather than treating each incident as an isolated event ^[12] — the same aggregation is what turns twenty individually interesting agent failures into a suite that covers a genuine failure taxonomy rather than twenty unrelated anecdotes.

The mechanics of harvesting from production have converged across the tooling vendors on a similar shape, worth stating as vendor practice rather than settled fact. LangSmith’s evaluation documentation describes converting production runs into dataset examples by capturing traces that received negative user feedback, flagging traces by heuristics such as unusually high latency or an execution error, and using an LLM to surface conversations worth a closer look ^[8]. Braintrust’s continuous-evaluation workflow answers the “how many is enough” question with a specific number rather than a feeling: it recommends waiting for at least one hundred facet summaries of a given failure classification before writing a scoring rule against it, so the pattern is established rather than a single unlucky trace being treated as a category ^[7]. These are documented product behaviors of specific commercial platforms, not universal laws, and a team building a suite in-house without such tooling can implement the same discipline directly: tag traces by outcome and by suspected failure class as they are triaged, and do not promote a tag to a permanent eval case until it has recurred enough times to be a pattern rather than an anecdote.

A second discipline sits underneath the harvesting: what gets pulled into the suite has to be balanced, not just plentiful. Anthropic’s guidance calls for including both cases the agent should complete and cases it should correctly decline or escalate, specifically to avoid a dataset that only rewards attempting a task and never rewards recognizing when it should not be attempted ^[1]. A suite built purely from failures over-indexes on the agent’s weak points and can miss regressions in what it already does well; a suite built purely from successes cannot distinguish a genuinely careful agent from one that has simply never been asked a hard question. The incident log supplies both halves if it is read for both: near-misses where the agent nearly went wrong and was pulled back, not only outright failures.

What counts as a pass when there is no single right answer

Code generation tasks have an unusually convenient property: a test suite either passes or it does not, and the grading problem mostly reduces to writing a good test suite. Most production agent tasks do not have that property. A customer-support agent’s reply, a research summary, or a multi-step operational action can be defensible in several different forms, and the question “did this pass” cannot be answered by string comparison.

Anthropic’s framework names three grader families and is candid about the trade-off each one makes. Code-based graders — string and regex matches, fail-to-pass and pass-to-pass test pairs, static analysis, direct verification of tool calls — are fast, cheap, objective, and reproducible, but brittle against any answer that is correct in a form the grader’s author did not anticipate. Model-based graders — rubric scoring, natural-language assertions, pairwise comparison, multi-judge consensus — are flexible and scale to ambiguous tasks, but need calibration against human judgment before they can be trusted. Human subject-matter review remains the gold standard for quality but is expensive and slow enough that it cannot grade every run ^[1]. None of the three dominates the others across every task; the practical answer, and the one both Anthropic and Braintrust converge on independently, is to combine them deliberately rather than pick one house style. Braintrust’s own documentation frames the same split as scorers that return a numeric quality score against classifiers that apply a categorical label, built from an autoevals library, an LLM-as-judge, or fully custom code depending on what the task needs ^[6].

The sharper practical question is not which grader family to use but how to know a chosen pass criterion is any good before it is trusted at scale. Anthropic’s stated bar is that a well-written task is one where two independent domain experts, looking at the same output, would reach the same pass or fail verdict ^[1]. That is a testable claim about the criterion itself, not about the agent — if it fails on human graders it will fail worse on an automated one. Hamel Husain’s methodology operationalizes this as an ongoing measurement rather than a one-time check: track the agreement rate between a model-based judge and a human evaluator on a held-out sample, run that comparison periodically rather than once, and prefer binary pass or fail labels over multi-point scales, because a five-point Likert scale is measurably more onerous to keep consistent across raters and across time than a clean yes or no ^[2]. OpenAI’s platform documentation makes the same argument from a different angle, warning specifically against “vibe-based” evaluation and observing that models discriminate between a small number of options — a pairwise comparison, a classification, a scored rubric — far more reliably than they generate an open-ended judgment from scratch, and recommending that criteria be structured accordingly rather than asking a judge to freely assess quality ^[5].

There is a genuine disagreement worth stating rather than resolving. Hamel Husain’s guidance leans toward keeping a human closely and continuously in the loop reading raw traces, treating automation as an accelerant for that reading rather than a replacement for it ^[2]. The evaluation platforms — Braintrust’s and LangSmith’s documentation both describe workflows where the majority of production traffic is graded by classifiers and judges, with humans confirming only flagged or sampled cases ^{[7] [8]}. These are not contradictory claims so much as different bets about where a scarce resource — expert attention — is best spent, and the right answer plausibly depends on production volume: a team handling dozens of cases a day can afford to read most of them personally; a team handling millions cannot, and needs the classifier layer to make the sampling itself intelligent. Treat this as an open trade-off to size against your own traffic, not a settled methodological question with one correct side.

Because a “pass” verdict is itself often a noisy measurement, not a fact, it is worth being explicit about how much noise a single run’s outcome carries before treating a change in the pass rate as real. If a task’s true pass rate is $p$ under a baseline configuration and a candidate change is worth detecting only once it shifts that rate by at least $\delta$, the number of repeated trials needed per configuration to detect the shift reliably — at significance level $\alpha$ and statistical power $1-\beta$ — is approximately

Plugging in a fairly ordinary case — a baseline pass rate of 70 percent, a minimum shift worth caring about of 10 percentage points, a 5 percent significance level, and 80 percent power — gives $z_{\alpha/2} \approx 1.96$ and $z_{\beta} \approx 0.84$, and the formula returns roughly 330 trials per configuration. That number is the whole argument for treating a single failed run, or even three failed runs out of five, with real suspicion rather than as proof of a regression: at ordinary sample sizes, most of what looks like a pass-rate change between two prompt versions is sampling noise, not signal. Teams that cannot afford several hundred repeated trials on every task for every change — nearly all of them — have exactly two disciplined options: accept a larger $\delta$ before declaring a regression, watching for a trend across several consecutive merges rather than a single run, or concentrate the expensive repeated-trial budget on the smaller set of tasks where a regression would be most costly and accept thinner evidence everywhere else. Either is defensible; treating a two-point pass-rate wobble on a twenty-task suite as a finding is not.

Wiring evaluation into the change pipeline itself

An eval suite that only runs when someone remembers to run it protects nothing. The organizational move that makes a suite actually load-bearing is treating it the way a codebase treats its test suite: it runs automatically on every change, and a failing run has a real consequence.

The Thoughtworks account of continuous delivery for machine learning, one of the earliest documented attempts to bring conventional CI discipline to model-based systems, describes a layered testing strategy spanning data validation, component integration contracts, and model-quality metrics evaluated against an established baseline, enforced through what the authors call threshold tests — an automated check that breaks the build if a quality metric crosses a defined line, functioning as a ratchet that a model is not allowed to slip back across ^[4]. That ratchet idea generalizes directly to an agent suite: a pass rate, once achieved, becomes the floor for every subsequent change, not a number that can silently drift downward one small regression at a time.

Concretely, this has become a documented feature of the current evaluation platforms rather than a bespoke pipeline each team reinvents. Braintrust describes running its evaluation suite as an automated check on every pull request, posting the per-case regression comparison directly on the request and blocking a merge when a score falls below a configured threshold ^[7]. LangSmith’s documentation draws a matching distinction between offline evaluation — curated datasets with reference outputs, run in a controlled setting specifically to catch regressions and compare versions before anything reaches users — and online evaluation, which runs against live traffic without a reference answer to catch anomalies a curated dataset never anticipated ^[8]. Braintrust’s own framing of the same idea splits the pipeline into three explicit stages — pre-deploy evaluation before a release, CI evaluation on every pull request, and continuous evaluation against live production traffic — with the last stage specifically responsible for catching production-only failures that no amount of pre-deployment testing anticipated ^[7]. That these are vendor product descriptions, not independent audits, should be kept in mind: they describe how each platform is designed to be used, not a controlled measurement of how well teams actually use it that way.

A harness detail worth calling out because it is easy to get wrong under time pressure: Anthropic’s roadmap for building an eval from scratch places “construct a robust harness” ahead of “design thoughtful graders,” specifically calling for an isolated environment with clean state for every trial, because a harness that lets state leak between trials — a file left over, a cache still warm — produces results that are not reproducible even when the grading logic is perfect ^[1]. This is the same discipline the Thoughtworks account calls a contract test between model and application, applied one level down: the environment the agent runs in during evaluation has to behave identically to a fresh one every single time, or the suite is measuring the leftover state of the previous run as much as it is measuring the change under test.

None of this is exotic engineering. It is the ordinary discipline of a CI pipeline for code, applied to a system whose outputs are graded rather than merely executed, and the vendors above have simply built products around making that discipline convenient rather than inventing a new one.

Pricing the suite before spending it

Every one of the practices above costs money and wall-clock time, and treating that cost as an afterthought is how eval suites get quietly abandoned six months after launch — not because they stopped being useful, but because nobody budgeted for what running them continuously would actually cost. The organizational fix is to make the spend explicit and tiered rather than uniform.

A concrete, independently reported number gives a sense of scale. The Holistic Agent Leaderboard project ran a standardized evaluation harness across nine models and nine agent benchmarks, producing 21,730 agent rollouts at a total reported cost of roughly forty thousand dollars — an average of close to two dollars per rollout at that scope of task and model — and had grown to 26,597 rollouts by April 2026, with the harness built specifically to parallelize execution across hundreds of virtual machines and cut evaluation time from weeks down to hours ^[11]. That is benchmark-grade thoroughness, not the cost of a single team’s daily regression run, but it is a useful anchor: full-suite, multi-model evaluation at real production-relevant scale runs to tens of thousands of dollars and days of wall-clock time even when heavily parallelized, which is exactly why almost no team can afford to run its complete suite on every commit.

The same research group’s earlier critique of agent benchmarking makes the underlying point directly: a narrow focus on accuracy without attention to cost produces evaluation regimes — and the agents optimized against them — that are needlessly complex and expensive, and the fix is to treat cost and accuracy as two coordinates that must be jointly optimized rather than a single number reported in isolation ^[10]. The leaderboard project’s own later finding sharpens this further with an uncomfortable result: across the majority of their runs, higher reasoning effort was associated with lower accuracy rather than higher, which means that spending more compute on an evaluation run is not a reliable way to buy a more trustworthy result, and a bigger eval budget is not automatically a better eval budget ^[11]. Both of these are the research team’s own reported findings rather than independently replicated results, and should be read with that caveat, but the design implication does not depend on the exact numbers: cost has to be measured and reported next to accuracy, not assumed to trend with it.

The practical answer most teams converge on is a tiered budget, and it is worth writing the tiering down as an explicit equation rather than an implicit habit, because an implicit habit is what quietly erodes into “we stopped running the full suite” a few months in. If every merge triggers a cheap smoke tier of $n_1$ tasks at unit cost $c_1$, run $k_1$ times a day, and a slower, more thorough tier of $n_2$ tasks at unit cost $c_2$, run only $k_2$ times a day, the daily evaluation bill is simply

The first term is the per-merge tax — it has to stay small enough that engineers do not start skipping it, which is the single most common way a CI gate quietly stops functioning. The second term is the trust budget — infrequent, thorough, and the number that should scale up before a release rather than on every keystroke. Braintrust’s continuous-evaluation documentation reports exactly this kind of tiering in its online-scoring guidance: score at somewhere between one and ten percent of traffic for high-volume applications, but score one hundred percent of traffic for flows judged critical enough that a missed regression is unacceptable ^[7]. That sampling rate is itself a budget decision made the same way — a cheap, wide net most of the time, and a full, expensive check only where the cost of missing something is highest. Writing the split down as two explicit numbers, rather than letting it happen ad hoc, is what keeps the fast tier fast enough to actually run on every change and the slow tier honest enough to actually catch what the fast tier cannot.

Why a suite goes stale, and how it gets gamed

An eval suite is not a one-time deliverable. Left alone, it decays in two distinct ways, and a team defending against only one of them is still exposed to the other.

The first is staleness by neglect: the production system keeps changing — new tools, new user populations, new failure modes — while the suite, written once against an earlier version of the system, keeps testing the same twenty or fifty cases forever. Anthropic’s own roadmap flags this directly, recommending that a team watch for saturation — a capability eval that has climbed to one hundred percent — as a signal that the task set has stopped being informative and needs harder cases, not as a signal of success to be celebrated and left alone ^[1]. A perfect score on an unchanged suite is at least as often evidence the suite stopped measuring anything as it is evidence the agent improved.

The second is more adversarial: a suite can be gamed, and not always by intent. A 2026 study out of UC Berkeley’s Center for Responsible, Decentralized Intelligence built an automated agent whose explicit purpose was to find shortcuts through eight widely used public agent benchmarks — including SWE-bench and WebArena — and reported near-perfect scores on most of them without the agent solving the underlying task at all ^[3]. The specific mechanisms are worth internalizing precisely because they generalize past the public benchmarks the researchers targeted. On SWE-bench, a small configuration file was used to hijack the test framework’s hooks so that every test outcome was rewritten as passing before the real grader ever inspected it. On WebArena, the agent simply read the task’s own configuration file over a local file path to retrieve the intended answer directly. On one benchmark, the validation function was found to accept an empty response as correct across every task in the set, because nobody had verified the checker actually rejected wrong answers. On a benchmark that used a model as judge, a hidden instruction embedded in the agent’s own output was enough to bias the judge toward a favorable score ^[3]. This is the research group’s own account of its own red-teaming exercise and has not, to my knowledge, been independently reproduced against every benchmark named — but the mechanisms described are specific and checkable against the named benchmarks’ public source, which is a different and stronger kind of evidence than an unverifiable claim would be.

None of those four mechanisms require a malicious agent. A sufficiently aggressive, reward-seeking optimization process finds exactly this kind of shortcut by accident, because from the agent’s perspective a hijacked test hook and a genuinely correct fix are both simply “the check passed.” The design lesson translates directly to a custom, internally built suite: the grading environment must be genuinely isolated from anything the agent under test can read or write, reference answers must never be reachable from inside the agent’s own execution environment even incidentally, and if a model is used as a judge, its input channel needs to be treated as adversarial — sanitized against injected instructions the way any other untrusted input would be ^[3]. Kapoor and colleagues’ earlier, less adversarial framing of the same failure mode is that inadequate holdout sets, or the complete absence of one, are a direct cause of agents taking shortcuts and overfitting to a benchmark’s specific quirks rather than solving the underlying task, and they argue for a held-out set the system’s own developers do not see and cannot iterate against ^[10].

The organizational answer to both failure modes is the same discipline, applied on a schedule rather than once: keep feeding fresh cases in from the ongoing incident stream described earlier in this piece, actively retire cases that have saturated or stopped discriminating between good and bad runs, and deliberately plant a small number of known-answer canary cases whose only purpose is to be checked periodically for whether the grading pipeline itself still rejects a wrong answer. Google’s ML production-readiness rubric, though written years before agentic systems existed, states the underlying principle in a form that still applies without modification: a test score for a production system is only useful as a rubric that is periodically re-measured and that assigns concrete, ongoing responsibility for keeping the tests themselves honest, not as a certificate earned once at launch and never revisited ^[9]. A suite is a piece of infrastructure with its own maintenance burden, and treating it as anything less is how it quietly stops protecting the system it was built for.

Predictions, and what would change my mind

These are forecasts, kept separate from the documented practices described above. Horizon: August 2028.

One. Tiered, cost-differentiated CI gating — a cheap smoke check on every commit paired with a slower, more thorough check on a longer cycle — will be documented as the default pattern by the mainstream agent-evaluation platforms, rather than teams continuing to run one uniform suite regardless of how large the change is. Disconfirmed if the leading eval platforms of 2028 still document only a single evaluation tier applied uniformly to every change.

Two. Adversarial self-audit of an organization’s own internally built eval harness — deliberately trying to find the shortcuts a Berkeley-RDI-style probe would find — will become a documented pre-launch step for custom suites, not only a technique applied after the fact to public leaderboards. Disconfirmed if mainstream practitioner guidance for building a custom eval suite in 2028 still omits any recommended adversarial self-audit of the harness itself.

Three. Production-trace-to-eval-case pipelines will account for the majority of how mature custom suites grow past their initial seed set, overtaking hand-authored task writing as the primary channel. Disconfirmed if 2028 guidance still describes hand-written task authorship, rather than triaged production traces, as the primary source of a suite’s growth beyond its first twenty to fifty cases.

Four. Mainstream eval tooling will begin surfacing an explicit confidence or repeat-count figure alongside every reported pass-rate delta, rather than presenting a single-run comparison as though it were a settled result. Disconfirmed if the major eval platforms of 2028 still report a bare pass-rate delta per merge with no indication of how many trials it rests on.

None of these requires a new capability breakthrough. They follow from where the discipline already visibly is: cost and accuracy already being tracked as a pair by the more careful teams, gaming already being demonstrated systematically against public benchmarks, and the tooling vendors already building, if not yet defaulting to, every piece described above.

What to take away

A custom evaluation suite is not a smaller, private version of a public benchmark. It is a different kind of artifact entirely — one grown from a specific system’s own incidents, graded by criteria two domain experts would actually agree on, wired into the same pipeline that ships every change, priced deliberately rather than by accident, and weeded on a schedule rather than left to rot. Every one of those is an organizational decision before it is a technical one: who reads the logs and how often, what threshold blocks a merge, how many dollars a day the fast tier is allowed to cost, who owns re-auditing the harness for the shortcuts a determined optimizer would find.

None of this is exotic. Postmortem culture, CI ratcheting, holdout-set discipline, and cost-accuracy joint optimization all predate agents by years, in some cases by decades. What is new is only the object being tested — a system that acts rather than merely answers — and the discipline required to test it well turns out to be the same discipline good engineering organizations already knew, applied without skipping the parts that are inconvenient to apply.