A robot does not get to pause the world while it thinks. A language model can take an extra second on a hard token; a robot arm reaching for a falling glass cannot. Everything in embodied AI that looks exotic from the outside — world models, sim-to-real transfer, diffusion policies — is downstream of one unglamorous constraint: the physical world runs on its own clock, and the robot’s software has to keep up with it or the mechanism fails in a way that a chatbot’s failure never does. This is a mechanics piece. It follows one control cycle from sensor to actuator, then widens out to how the policy running inside that cycle got trained, how it survives the move from simulation to hardware, and what it costs to keep it running safely in the field.
The loop that has to close every cycle
A fielded manipulation or locomotion policy runs a perception-action loop at a fixed rate — commonly somewhere between 10 Hz for a high-level planner and several hundred Hz for a low-level joint controller, with the two often nested. Inside one cycle the system has to read sensors (cameras, joint encoders, force-torque sensors, in some cases motion-capture or IMU data), run inference through a neural network policy, and issue actuator commands, all before the next cycle’s sensor read arrives. Miss that budget consistently and the controller is not degraded — it is unstable, because contact and balance are physical processes that keep evolving whether or not the software has produced a decision.
This is why vision-language-action (VLA) models, the architecture behind systems like RT-2, are a genuinely different design problem from a text-only language model even when they share the same transformer backbone. RT-2 co-fine-tunes a large vision-language model on both web-scale image-text data and robot trajectory data, expressing actions as discretized text tokens so the same next-token machinery that predicts words also predicts joint or end-effector commands [1]. The payoff the paper reports is transfer of web-scale semantic knowledge into the control policy — the robot can act on instructions and object categories not present in its own demonstration data, including a reported ability to reason about which object is smallest or which is closest to another, because that reasoning is inherited from the underlying vision-language pretraining rather than from robot data alone [1]. That is a specific, checkable claim about generalization on the paper’s own evaluation tasks — it is not a claim that the same model would generalize equally well on tasks or objects the authors did not test, and the paper itself frames the improvement as measured against baseline policies on their benchmark, not as an unqualified capability.
The cost of that generalization is inference latency, and it lands exactly where the loop is least forgiving: at the interface between a large model’s forward pass and a control cycle measured in milliseconds. This is one reason production robot stacks typically separate a slow, semantically rich planner from a fast, low-level controller — the VLA or world-model layer proposes a goal, waypoint, or short action chunk at a low rate, and a separate, much cheaper feedback controller closes the loop against real-time disturbances in between. The architecture question in embodied AI is rarely “can a big model produce a good action” and almost always “how do you keep a big model’s latency out of the part of the loop that cannot wait for it.”
World models: rehearsing the future instead of measuring it
A perception-action loop only works if the policy has some model, explicit or implicit, of what its action will do to the world. One family of approaches makes that model explicit and learned: a world model is a neural network trained to predict future latent states (and often future observations) conditioned on actions, and a policy is trained by rolling that model forward in imagination rather than by acting in the real environment on every training step.
Dreamer-style world models learn a compact latent-dynamics model — a recurrent state-space model that predicts both a stochastic and a deterministic component of future latent state — and then train a policy by backpropagating through imagined rollouts inside that latent space, rather than through real or simulated environment steps directly [8]. DayDreamer took that specific mechanism, previously demonstrated mostly in video-game and simulated benchmarks, and applied it to physical robots learning online, without a simulator and without human demonstrations, reporting that four different physical robots (a quadruped and several manipulator arms) learned locomotion and manipulation skills directly from real-world interaction using the same world-model architecture across all of them [8]. The significant claim here is narrower than “world models let robots learn in the real world” in general — it is that this specific latent-imagination training loop, previously validated in simulation, transferred to real hardware without a simulator in the authors’ own experiments, on the specific tasks and robots they report.
The equation names the actual asymmetry a world model buys: the policy is optimized against
Sim-to-real: training somewhere the mistake is free
Most learned locomotion and a large share of learned manipulation policies today are trained primarily in physics simulation, for the plain reason that a simulator can run thousands of times faster than real time, in parallel across many instances, and a bad training-time policy in simulation costs nothing when it falls over. The unresolved problem this creates is the “reality gap”: a simulator’s contact model, actuator dynamics, and sensor noise are all approximations, and a policy overfit to those approximations can fail on hardware that behaves even slightly differently.
Domain randomization is the standard mitigation, and it is a specific, well-defined technique rather than a synonym for “make the simulator noisy.” Peng et al. randomize the simulated dynamics — mass, friction, and other physical parameters — across training episodes, so the policy is optimized against a distribution of plausible dynamics rather than one fixed, exactly-specified simulator, on the premise that real-world dynamics will fall somewhere inside (or close to) that trained distribution [7]. Their reported result is transfer of a robotic pushing-task policy to physical hardware with no additional real-world training, using dynamics randomization alone, on the specific manipulation task the paper evaluates [7]. It is a demonstration that the technique can close a reality gap on a chosen task, not a general guarantee about the size of gap it can close on an arbitrary one, and later theoretical work on domain randomization frames the transfer bound explicitly in terms of how well the randomization distribution’s tunable parameters actually bracket the unknown real parameters — a bound that can be loose or tight depending on how the randomization range was chosen, not a fixed constant.
Legged locomotion has produced some of the cleanest published demonstrations of this pipeline because a quadruped’s failure mode — falling — is immediate, visible, and does not require semantic labeling to score. Hwangbo et al. trained a neural network policy for the ANYmal quadruped using reinforcement learning in simulation, including a learned actuator model to bridge part of the sim-to-real gap in how the physical motors actually respond to commands, and report the resulting policy running on hardware with locomotion skills — including recovery from falls in configurations not explicitly designed into the controller — exceeding what earlier model-based controllers on the same robot achieved [5]. A follow-up extended the same lab’s approach to natural, unstructured terrain by training the controller on proprioceptive feedback alone (joint encoders and inertial sensing, no camera), and reported zero-shot transfer from simulation to a range of real outdoor environments the policy was not specifically tuned for [6]. “Zero-shot” here is a specific technical claim — no additional training or fine-tuning on the target terrain — not a claim that the policy handles all terrain equally well; both papers report specific test conditions and specific failure boundaries alongside their successes.
Manipulation: policies that have to handle contact, not just motion
Legged locomotion mostly has to keep a rigid body upright; manipulation has to make and break contact with objects whose mass, friction, and compliance are only ever partially known, and it has to do so under a much higher-dimensional and often multimodal action space — there are frequently several equally valid ways to grasp or place an object, not one correct trajectory.
Diffusion Policy reframes visuomotor manipulation as a conditional denoising diffusion process over action sequences rather than a direct action regression: instead of predicting one action (or one Gaussian over actions) per observation, the policy learns to iteratively denoise a randomly sampled action-sequence proposal into a plausible trajectory, conditioned on the current visual observation [3]. The stated motivation is exactly the multimodality problem above — a single-mode regression policy trained by imitation learning tends to average between multiple valid demonstrated solutions and produce an invalid blended trajectory, while a diffusion model’s iterative sampling process can represent distinct modes. The authors report a 46.9 percent average improvement over prior state-of-the-art imitation-learning methods across 15 tasks drawn from four existing manipulation benchmarks [3] — a benchmark-relative number tied to those specific tasks and baselines, not a universal multiplier on manipulation performance.
Behind every learned manipulation policy is the demonstration data it was trained on, and collecting that data at any useful scale has become its own engineering problem. Mobile ALOHA is a low-cost whole-body teleoperation rig: a human operator controls a mobile, bimanual robot’s base velocity and both 14-degree-of-freedom arms together, so that mobile manipulation demonstrations — not just tabletop pick-and-place — can be collected directly by a human driving the real hardware [4]. The paper’s specific empirical finding is that co-training on an existing, larger static-manipulation dataset alongside a much smaller set of newly collected mobile demonstrations increased task success rates by as much as 90 percent relative to training on the small mobile dataset alone, on the household tasks they tested [4] — evidence for data co-training as a way to make a scarce, expensive data source (whole-body mobile demonstrations) go further, not a claim that any two datasets combine this favorably.
At a larger scale, the Open X-Embodiment collaboration pooled robot trajectory data across 21 institutions into a single standardized dataset spanning 22 different robot embodiments and over a million real trajectories, and trained RT-X models on the combined set [2]. The reported result is positive transfer: RT-1-X, trained across embodiments, outperformed the same architecture trained only on a single robot’s own data by roughly 50 percent in the low-data regime for that robot [2]. That is a claim about data efficiency for embodiments already represented in a large pooled dataset — it does not, on its own, establish how the same approach behaves for a robot morphology genuinely absent from the pool.
Safety: a measured standard, not a general disposition
None of the preceding mechanisms matter to a deployment unless the resulting robot is safe to operate near people, and that question has a specific technical standard behind it rather than a general engineering ethos. ISO/TS 15066 (since folded into ISO 10218-2:2025) specifies collaborative-robot safety requirements building on ISO 10218, and it defines four distinct collaborative operating modes: safety-rated monitored stop, hand guiding, speed-and-separation monitoring, and power-and-force limiting [9]. Power-and-force limiting in particular is defined quantitatively — the standard sets reference values for the maximum force and pressure a robot may transmit to specific regions of the human body in a collision, values manufacturers use to calibrate torque limits and contact detection thresholds on their hardware [9]. Speed-and-separation monitoring is a different mode entirely: rather than limiting contact force, it uses sensing to maintain a minimum protective separation distance, reducing robot speed or stopping it as a person approaches, which is the mechanism figured above at the mesh-guarded cell.
The important qualification, stated plainly: certification to this standard governs a specific, defined set of collaborative behaviors under specified conditions — payload, speed, and the specific body-region force limits it enumerates. It is not a general certification that a learned policy is safe to run near people in an open-ended task; a policy trained on the methods above still needs its own task-specific risk assessment, because the standard constrains the robot’s physical behavior, not the correctness of an arbitrary learned decision inside that behavior.
Deployment economics: what the loop actually costs
Put the pieces together and a fielded embodied-AI system’s cost structure has three separable components, each traceable to a mechanism above. Data collection cost scales with the demonstration-gathering method: whole-body teleoperation rigs like Mobile ALOHA require a human operator’s time per demonstration, which is why cross-embodiment data pooling and co-training, as in Open X-Embodiment, are attractive — they amortize a large existing corpus against a much smaller new-robot data collection budget rather than requiring a full new dataset per robot [4, 2]. Simulation cost scales with how much of the reality gap has to be closed by randomization breadth versus by real-world fine-tuning; wider randomization ranges generally demand more simulated training compute and more careful tuning of the randomization distribution itself, per the theoretical framing of the sim-to-real bound discussed above [7]. Inference cost at the edge is shaped by the split between a slow semantic layer (a VLA or world model) and a fast low-level controller — a design that lets a robot afford a large model’s latency only where the control loop can tolerate it [1, 8].
Scenario, not fact: a plausible near-term deployment pattern is that manipulation tasks with well-characterized contact (rigid parts, controlled fixtures) convert to learned policies faster than tasks involving deformable or unfamiliar objects, because diffusion- and imitation-based policies still depend on demonstration coverage of the specific contact modes involved [3]. This is an extrapolation from the mechanisms above, not a documented industry trend, and it would be disconfirmed by evidence that deformable-object manipulation policies are being fielded in production at comparable rates to rigid-part manipulation within the next two to three years, or by benchmark results showing diffusion-based policies generalizing well outside their demonstrated contact modes without additional data.
Prediction, horizon 2029, explicit and falsifiable: cross-embodiment pretraining in the style of Open X-Embodiment will most likely become the default starting point for new manipulator deployments — rather than training each new robot’s policy from scratch — given the reported low-data transfer gains already measured for embodiments already in the pool [2]. Assumption: institutions continue depositing standardized trajectory data into shared pools at roughly the current or faster rate. Observable indicator: a majority of new commercial manipulation-policy papers citing a pooled, multi-institution dataset as their pretraining source rather than only single-robot data. Disconfirmation condition: if by 2029 the majority of published commercial deployments still train exclusively on single-robot, in-house data with no cross-embodiment pretraining step, the prediction is falsified.
Where the analogy to language models breaks down
It is tempting to describe embodied AI as “language models for the physical world,” and the vision-language-action architecture behind RT-2 invites exactly that comparison, since it reuses a transformer trained on internet-scale text and images [1]. The analogy is useful for one thing and misleading for almost everything else. It is useful because it explains where the semantic generalization comes from: a policy that has seen millions of captioned images of everyday objects can recognize and reason about an object category it never saw in robot demonstration data, which is the specific mechanism RT-2’s authors point to for their reported gains [1]. It is misleading because a language model’s output is graded by whether the next token is plausible, evaluated after the fact, while a robot’s output is graded by whether a physical joint torque, applied now, keeps a body upright or a grasp closed against gravity, evaluated by physics in real time. There is no equivalent in text generation to a diffusion policy needing to represent several genuinely different, physically valid grasp trajectories rather than collapsing to their statistical average [3] — the analogous failure in text generation (a bland, hedged answer) is merely unsatisfying; the analogous failure in manipulation (an averaged, invalid trajectory) drops the object.
This also explains why progress in the two fields is not fungible. Scaling a language model’s parameter count and pretraining corpus has produced fairly reliable, roughly monotonic gains on held-out language benchmarks. Scaling a robot policy’s parameter count without also scaling the diversity of physical dynamics it has been exposed to — through domain randomization, cross-embodiment data, or real-world fine-tuning — does not reliably produce the same gains, because the bottleneck is frequently the reality gap rather than model capacity [7]. That is the specific reason Open X-Embodiment’s contribution is framed around data diversity across embodiments rather than around a larger model trained on one robot’s own data [2].
What the mechanics actually establish
None of the individual results above should be read as a general solution to embodied intelligence. Each is a specific technique validated on specific tasks, specific robots, and specific benchmarks, with its own reported failure boundary. RT-2 improves generalization on the tasks its authors evaluated; diffusion policy’s improvement is relative to specific baselines on specific benchmarks; ANYmal’s zero-shot terrain transfer was tested on specific terrain types; Open X-Embodiment’s transfer gains are reported for embodiments already represented in its dataset. What the mechanics do establish, cumulatively, is a coherent explanation for why embodied AI has advanced along a specific axis — training at scale somewhere cheap (simulation, pooled data, imagined rollouts) and transferring carefully to somewhere expensive and unforgiving (the real control loop) — rather than along the axis language models advanced on, which was mostly just more data and more parameters in one training regime. The perception-action loop is the constraint that makes that transfer non-optional; everything else in this article is a strategy for surviving it.
