A policy that acts on matter
From inside the network, generating a sentence and closing a gripper look like the same problem: a conditional distribution over the next action given everything observed so far. The architectures are converging on exactly that view, and the current generation of robot policies is built from pretrained vision-language backbones with an action head bolted on [3, 4]. The similarity is real. It is also the source of most of the overconfidence in the field, because it hides the one difference that governs everything else.
A language model’s environment is a buffer it owns. Nothing in that buffer has a temperature, a coefficient of friction, or a prior history the model did not write. When the model is wrong, the wrongness is a token, and the standard remedy — sample again, or let the model notice and correct itself in a later paragraph — costs a fraction of a cent and leaves no residue.
A robot’s environment is matter. It carries state the policy did not create, cannot inspect, and cannot revert. The mug that was knocked off the counter is not un-knocked by a better second attempt. This is not a quantitative difference in difficulty. It changes which algorithms are admissible, which data can exist, what a benchmark number means, and what “reliability” has to be worth before deployment is rational.
Six constraints follow from it, and each of them is structural rather than a temporary engineering deficit. This article works through them in order, then asks what the recent large behaviour models have actually established.
There is no corpus, and collecting one requires the robot
The first constraint is the one most often waved away with the phrase “we just need more data”.
Language models were trained on a corpus that already existed. Nobody commissioned the internet. Robot manipulation has no such windfall, because a trajectory is a record of a specific body acting through a specific controller, and there is no archive of joint torques and gripper states lying around waiting to be scraped. Every datum has to be produced by moving a real machine through the real world, and the machine you need is the machine you are trying to build.
The scale of what this actually yields is worth stating precisely, because it is routinely inflated. The Open X-Embodiment collaboration pooled the field’s holdings into one standardised release, and its own description of the result is a dataset drawn from 22 different robots across 21 institutions, covering 527 skills over 160,266 tasks [1]. DROID, one of the more ambitious purpose-built collections, reports 76,000 demonstration trajectories amounting to 350 hours of interaction, gathered across 564 scenes and 84 tasks by 50 human collectors working in three continents over twelve months [2]. Physical Intelligence describes its own pre-training mixture as over 10,000 hours of robot data — 903 million timesteps drawn from its own fleet plus portions of the open collections, spanning 7 robot configurations and 68 tasks — and characterises it as by far the largest robot learning experiment measured in quantity of robot data [3]. That is a vendor’s characterisation of its own effort, and I have no way to audit the underlying counts; but it is a fair upper bound on what the field has assembled.
Hold those numbers next to a text corpus and the asymmetry is not one of degree. Fifty people working for a year produced 350 hours. A video platform accumulates more than that every few seconds. And the robot numbers do not decompose the way text does: 350 hours on Franka arms in 564 rooms tells you comparatively little about a different gripper, a different wrist camera placement, or a different controller. Embodiment is a covariate that appears in every sample and does not average out.
There is a second, sharper version of the problem. The trajectories that matter most are the ones a working policy would generate — the recoveries, the regrasps, the moment a screw crossthreads and the wrist backs off. Those states are precisely the ones a teleoperator rarely enters and a mediocre policy cannot reach safely. The distribution you need to learn from is downstream of the competence you are trying to acquire. Kroemer, Niekum and Konidaris frame this as the structural difficulty of manipulation learning: the state space is high-dimensional, the dynamics change discontinuously with contact, and the useful abstractions are task-dependent rather than given [13].
What domain randomisation buys, and what it cannot
Simulation looks like the obvious escape, and for one class of problem it genuinely is.
Tobin and colleagues established the canonical result in 2017. By randomising the rendering — textures, lighting, camera pose, distractor objects — they trained an object detector entirely on non-realistic simulated RGB images that localised real objects to 1.5 centimetres, with no real images used in training, and used it to grasp in clutter [6]. The insight generalised: if the simulator’s variation is wide enough, reality becomes one more sample from the training distribution rather than a distinct domain.
The extension to dynamics produced the field’s most cited demonstration. OpenAI’s in-hand reorientation work randomised physical properties including friction coefficients and object appearance, and transferred a policy trained entirely in simulation onto a physical Shadow Dexterous Hand, where behaviours such as finger gaiting and controlled use of gravity emerged without demonstrations [7]. Two numbers from that same paper are more instructive than the headline. First, the price: learning to rotate the object in an unrandomised simulator required about 3 years of simulated experience, while reaching comparable performance under full randomisation required about 100 years [7]. Randomisation is not free regularisation; it is a roughly thirty-fold compute multiplier. Second, the residue: the same policy achieved a median of 50 consecutive successes in simulation and a median of 13 on the physical hand [7]. That gap survived the randomisation that was specifically designed to close it.
This is the part that is generally elided. Domain randomisation can only randomise parameters the simulator already has. It buys robustness to variation within a model. It buys nothing against the model being the wrong shape. Randomising a friction coefficient over a wide range does not help if the underlying contact law does not describe how the real surface behaves at all.
Contact is where the simulator stops being a simulator
Contact is not one hard sub-problem among several. It is the specific place where the physics engine’s abstraction and the world diverge, and it happens to be the abstraction on which all manipulation rests.
Rigid-body contact models are approximations with known pathologies. Fazeli and colleagues performed system identification on six commonly used planar contact models and compared them against experimental impact data, establishing performance bounds and documenting salient failure modes — including the finding that the fitted parameters are difficult to give any physical interpretation, so a “friction coefficient” identified from data is often a fitting constant rather than a measurable property of the surface [8]. That result should change how one reads a sim-to-real pipeline: tuning the simulator to match reality can succeed numerically while the tuned quantity means nothing transferable.
Acosta, Yang and Posa tested this directly at the simulator level, identifying contact parameters for Drake, MuJoCo and Bullet and evaluating whether each reproduced measured real-world impact trajectories. Their finding was asymmetric: the simulators capture inelastic impacts well but fail to capture elastic impacts, and on a Cassie biped jumping and landing they reproduced the bulk motion while accuracy remained limited by numerous model differences between the real robot and the simulators [9]. Bulk motion transfers. The details of energy return, stiction and micro-slip at the interface do not.
Deformable objects remove the abstraction altogether. A survey of the area locates the difficulty in the infinite dimensionality of a deformable object’s state space and the complexity of its dynamics, which together defeat the state representations that rigid-body manipulation relies on [10]. There is no pose. A folded towel does not have six degrees of freedom that a perception stack can estimate; it has a configuration that is partly self-occluded and only partly recoverable from any external view. Every technique that assumes an object state can be estimated, tracked and planned against loses its footing here — which is inconvenient, because a large share of the tasks people actually want robots to do involves cloth, food, cable, foam, paper or skin.
The world has no undo
The third constraint is the one with the cleanest formal statement and the widest consequences.
Reinforcement learning as normally practised assumes the environment can be reset. Eysenbach and colleagues made the assumption explicit precisely in order to attack it: real-world learning means repeatedly attempting a task and resetting between attempts, not all tasks are easily or automatically reversible, and in practice the process therefore requires extensive human intervention. Their proposal — learn a reset policy alongside the forward policy, and use its value function to detect when the forward policy is about to enter a non-reversible state, triggering an uncertainty-aware safety abort — is an admission of how binding the constraint is [11].
Retrying is not merely expensive in the physical world; for some states it is undefined. A text model that produces a bad paragraph can be sampled again from an identical starting state. A robot that has shattered a plate cannot be. The starting state is gone, and the policy’s own error has moved the environment somewhere the policy was not trained for.
This compounds badly over long horizons. If a task requires
The independence assumption is generous — real failures correlate, because one slip puts the system into a region where subsequent steps are also more likely to fail — but even the generous version is brutal. For a 300-step manipulation sequence to complete nine times in ten, per-step reliability must be about 0.99965. A per-step reliability of 0.99, which would be an excellent number for almost any perception module, yields a task success rate of roughly five percent. This is straightforward arithmetic rather than a finding from any cited study, but it explains an otherwise puzzling pattern: policies that look near-perfect on individual skills collapse on chained ones, and the collapse is not a bug to be found, it is the exponent.
Deliberation has a deadline
The fourth constraint is the one that most directly forecloses the strategy that has worked so well for text.
Language models became substantially more capable by thinking for longer at inference time. That option is bounded on a robot, and the bound is not a matter of taste. A closed-loop controller running at frequency
The pi-zero model is documented as producing actions at up to 50 Hz for dexterous tasks [3], which leaves a 20-millisecond budget for everything. No amount of chain-of-thought fits in 20 milliseconds. Worse, missing the deadline is not merely slow: a controller that arrives late in a contact-rich interaction is acting on a stale world model, and in a stiff contact the resulting force error can be large enough to damage the hardware or the object.
The field’s structural answer is action chunking: predict a block of future actions in one forward pass and execute it open-loop while the network runs again. Zhao and colleagues introduced this explicitly to combat compounding error in imitation learning, and reported 80 to 90 percent success on six fine manipulation tasks — including opening a translucent condiment cup and slotting a battery — from around 10 minutes of demonstrations per task [12]. The pi-zero architecture uses the same device, emitting chunks of 50 future actions [3].
Chunking is a real advance and it is also a trade, which the enthusiasm around it tends to obscure. Executing a chunk open-loop means the policy is deliberately blind for the duration of the chunk. Longer chunks buy more deliberation time per decision and lose more reactivity to whatever the world does in the interval. Analysis: this places a ceiling on the “just add more test-time compute” strategy that is set by the task’s own dynamics, not by hardware. Tasks whose disturbances are slow can afford long chunks; tasks involving slipping, tipping or a human moving nearby cannot, and no future accelerator changes that.
Seeing an object is not knowing it
The fifth constraint is a partial-observability problem that gets misfiled as a perception problem.
Modern perception recovers geometry and semantics well. It does not recover the quantities the controller actually needs. Mass, centre of mass, inertia, the friction coefficient of a surface, the stiffness of a joint, the compliance of a grip — none of these are visible. Dutta, Burdet and Kaboli address exactly this set, building an active visuo-tactile framework to infer shape, friction coefficient, mass, centre of mass and inertia through non-prehensile pushing, with the system selecting which exploratory push to make next [15].
The structure of that work is the point. The parameters had to be acted for. You cannot look at a closed cardboard box and know whether it holds books or packing foam; you have to nudge it and observe what happens. This makes manipulation an information-gathering problem in which the information-gathering actions are themselves risky and irreversible. The push that tells you the object is heavier than expected may also be the push that topples it.
For a learned policy, the practical consequence is that visually identical situations require different actions, and the discriminating variable is absent from the observation. No amount of visual training data resolves it. This is a strong argument that tactile and force sensing are not an optional enhancement but a requirement for the contact-rich portion of the task space — and correspondingly, it is a reason to discount demonstrations of manipulation conducted entirely from cameras.
The long tail is the environment itself
The sixth constraint is the ordinary world, and it is best appreciated by looking at what a serious attempt to enumerate it produces.
BEHAVIOR-1K defines 1,000 everyday activities grounded in 50 scenes — houses, gardens, restaurants, offices — with more than 9,000 objects annotated with physical and semantic properties, and its authors state plainly that these activities are long-horizon and dependent on complex manipulation skills, both of which remain a challenge for even state-of-the-art robot learning solutions [14]. Notably, the activity list was derived from a survey asking people what they actually want robots to do, rather than from what is convenient to simulate [14].
A thousand activities across fifty scenes is an enormous curatorial effort and it is still a rounding error against the real distribution. Kitchens differ in counter height, cabinet hardware, lighting, floor friction and the particular way a given household’s drawer sticks. A benchmark samples the head. Deployment samples the tail, and the tail is where irreversibility, unmodelled contact and unobservable physical parameters all arrive at once.
What the large behaviour models have and have not shown
Recent progress is genuine, and the claims around it need careful sorting into three bins: what a developer states about its own system, what an external evaluation measured, and what neither establishes.
Developer claims. Google DeepMind reports that Gemini Robotics executes reactive movements across a range of manipulation tasks, is robust to variation in object type and position, handles unseen environments, follows open-vocabulary instructions, and after fine-tuning can learn new short-horizon tasks from as few as 100 demonstrations and adapt to novel embodiments [4]. Physical Intelligence reports that pi-zero, trained on its 10,000-hour mixture, performs complex multi-stage tasks including laundry folding, table cleaning and box assembly across single-arm, dual-arm and mobile platforms [3]. These are the developing laboratories’ own accounts of their own systems, in technical reports rather than peer-reviewed venues, and they should be read as claims. They are also, on their face, substantial: 100 demonstrations for a new short-horizon task is a very different regime from training a policy per task per robot per room.
External measurement. RoboChallenge ran released policies on a standardised real-robot benchmark called Table30, comprising 30 tasks on a fixed table across four robot platforms, with 10 rollouts per task. The reported average success rates in the task-specific setting were 43.7 percent for pi-zero-point-five, 28.3 percent for pi-zero, and 11.7 percent for CogACT; the generalist variant of pi-zero-point-five, evaluated without per-task specialisation, averaged 17.7 percent [5]. The authors note that temporal dependence and soft-body manipulation remain particularly hard, with success rates substantially below simple pick-and-place [5].
Those two paragraphs are not in contradiction, and it is worth being precise about why. The demonstration videos are real; the benchmark numbers are real; they measure different things. A video shows that a policy can complete a task under conditions the developer selected. The benchmark reports how often it does, on tasks and hardware it was not tuned for, averaged over repeated trials. The drop from 43.7 to 17.7 percent between the task-specific and generalist settings of the same model family is the single most informative number in the set, because it isolates the cost of not specialising — and generality is the entire premise of the large-behaviour-model programme.
I hold this evidence with an explicit caveat: an external harness cannot guarantee that every policy was configured as its authors intended, and calibration disputes of this kind are normal and unresolved. That is itself the point. There is no equivalent in robotics of a widely agreed held-out test set, and until there is, cross-system comparison rests on benchmarks whose construction the compared parties did not agree to in advance.
Where informed people genuinely disagree is on what the remaining gap is made of. One camp holds that it is a data problem: the scaling curve simply has not been walked far enough, and fleets collecting continuously will get there. Another holds that it is a representation problem, that behaviour cloning from teleoperation cannot produce recovery behaviour it has never observed, and that something closer to real-world reinforcement learning or model-based control is required. A third holds that the binding constraint is hardware — that human-comparable tactile sensing and compliant, backdrivable actuation are prerequisites no policy can substitute for. The evidence assembled here is consistent with all three, which is precisely why it should not be reported as settled.
Predictions, with the observations that would falsify them
These are forecasts, separated from the sourced analysis above. Horizon: 8 August 2029. Assumption: no discontinuity in actuator or tactile-sensor cost.
One. Reported success rates on standardised, externally administered real-robot benchmarks will remain below 80 percent averaged across tasks involving deformable objects, even as rigid pick-and-place saturates. Disconfirmed if an independently administered benchmark with at least 10 trials per task reports above 80 percent on a deformable-object suite.
Two. The gap between task-specific and generalist evaluation of the same model family will narrow but not close, remaining above 10 percentage points. Disconfirmed if a published external evaluation shows a generalist policy within 10 points of its specialised counterpart across a 20-plus-task suite.
Three. Commercial deployments will concentrate in environments engineered to remove the tail — fixed lighting, known fixtures, controlled object sets — rather than in unmodified homes. Disconfirmed if a robot performing unstructured household manipulation ships at consumer volume with published, independently verified reliability figures.
Four. Tactile and force sensing will move from research accessory to standard component on manipulation platforms, because the unobservable-parameter problem does not yield to vision. Disconfirmed if the leading published policies in 2029 remain vision-only and match force-equipped systems on contact-rich benchmarks.
None of these requires the field to stall. They follow from the constraints above being structural: data that must be produced by the machine being built, simulators that model bulk motion but not interfaces, an environment with no undo, a control loop with a hard deadline, and physical parameters that must be acted for rather than seen.
What to take away
The embodiment gap is not a gap in model quality. It is the distance between a decision and its effect on matter — a compliant, coupled, lossy transmission in which intention arrives changed and the change cannot be taken back.
When a demonstration video crosses your desk, ask four questions. How many trials, and who ran them. What was the starting state, and who arranged it. What happens on the attempt after a failure. And which of the objects were rigid. A system that cannot answer those has shown you that a thing is possible, which is worth something, but it has not shown you that the thing is reliable — and in a domain where errors are not retryable, reliability is the only property that converts a demonstration into a deployment.
