Tagged evaluation · show all articles
Image-text systems tokenize a grid and align two vectors. Video is a spacetime volume, audio is a waveform, and a 3D scene is a continuous field — none of them arrives as a grid, and each forces its own discretization scheme before any model can read it.
A model that accepts an image is not the same as a model that uses one. A working guide to the decisions that separate the two: API or adapter, what the training pairs must contain, how to test for the unimodal shortcut, and what the tokens actually cost.
A feature-visualization narrative that looks right and a circuit that has been checked against a known answer are different kinds of evidence. Most published claims sit closer to the first than the coverage of them usually admits.
A single successful run shows a system can succeed once. Pass@k, pass^k, and repeated-trial statistics are how anyone finds out whether it succeeds reliably — and task length changes the answer.
AIME, GPQA and FrontierMath test different things by different rules. Reasoning effort and extended thinking are documented settings, not fixed traits, and a right answer does not certify the reasoning behind it.
Retrieval-augmented generation fails quietly. Ten distinct, documented mechanisms, from a stale index to a fabricated citation, account for nearly every expensive production RAG incident, and each has its own detection and its own fix.
Two axes decide what retrieval-augmented generation becomes by 2035: whether evidence-gathering stays a separate, inspectable step, and whether what it returns can be verified. Crossing them gives four futures, and current evidence does not yet pick one.
A language model's mistakes are cheap because its environment is a buffer it owns. A robot's environment is matter with state it did not write and cannot roll back, and every hard part of embodied AI follows from that.
Reinforcement learning from human feedback does not install correctness. It fits a scalar model of what raters approved of and then pushes a policy up that surface — which is a different thing, in identifiable places.
A starter culture is a population propagated from itself. It runs indefinitely if it is fed and managed, and it drifts or dies if it is only ever fed its own unmanaged output. Synthetic training data behaves the same way.