#distribution shift

Tagged distribution shift · show all articles

A row of rack-mounted server blades running isolated agent sandboxes, one blade's activity light still lit and its drive tray partly drawn while the blades either side sit dark and fully seated

The Hardest Unsolved Problems in AI Agent Evaluation and Reliability

A benchmark score describes a trajectory that has already ended. The hardest problems in agent evaluation are about the ones still running, the ones nobody re-ran, and the ones no aggregate number shows.

By Brecht Corbeel · August 26, 2026 · 15 min read

Browse by Category

Surprise me →

All AI Hardware & Semiconductors 100 AI Agents & Systems 42 Foundation Models 30 Model Evaluation 22 AI Industry 12 AI Infrastructure 11 Edge AI & Electronics 9 AI Safety 9 AI Research 9 Open Models 8 History of Technology 5 Invention & Engineering 3 Industry Analysis 2 Datacenters & Infrastructure 2 Writing & Information 1 Structural Biology 1 Small & Edge Models 1 Semiconductor Markets 1 Security & Technology 1 Security 1 Scientific Revolution 1 Robotics & Embodied AI 1 Quantum Information 1 Planetary Science 1 Physics 1 Particle Physics 1 Origins & Astrobiology 1 Open-Weight Models 1 Neuroscience 1 Metrology 1 Mathematics & Computation 1 Materials Science 1 Labour & Technology 1 Interpretability 1 Institutions & State Capacity 1 Institutions & Economy 1 Inference Economics 1 Industrial Revolutions 1 Humanity & Technology 1 History of Computing 1 Governance & Policy 1 Evolution as Mechanism 1 Evolution & Ecology 1 Ethics of Technology 1 Energy & Civilization 1 Earth & Climate 1 Demography & Urbanization 1 Data & Training 1 Cumulative Culture 1 Cosmology 1 Condensed Matter 1 Chemistry & Catalysis 1 Alignment & Safety 1 Agriculture & Cities 1 Agent Evaluation 1 AI Security 1 AI Economics & Systems 1 AI & The Web 1