Tagged LLM-as-judge · show all articles
An agent can behave exactly as trained, a benchmark can be built in good faith, and the reported number can still be wrong, because the failure was never in the run. It was in how the evaluation itself was designed, graded and counted.
Public benchmarks measure someone else's agent on someone else's tasks. A suite that actually protects your system has to be grown from your own incidents, your own pass criteria, and a compute budget you set on purpose.
Two questions decide whether agent evaluation becomes a certified, externally audited discipline by 2035, or stays a signal vendors mostly produce and grade themselves. Four scenarios, each with a falsifier.
A retrieval score and an answer-quality score are counted by different instruments, on different objects, at different points in the pipeline. Either one can look excellent while the other is quietly broken, and most reported RAG numbers do not say which.