Foundation models for robotics are large models pretrained on broad data, such as images, text, and many robots' demonstrations, that can be adapted to control different robots across many tasks, instead of training one narrow model per task.
A foundation model, as defined by Stanford researchers in 2021, is trained on broad data at scale and adapted to many downstream tasks. In robotics, the idea is to apply this pattern to control. A notable example is RT-2 (2023), a vision-language-action (VLA) model that fine-tunes a vision-language model on robot data and outputs robot actions as tokens, like words.
Data is the bottleneck. Robot demonstrations are expensive to collect. The Open X-Embodiment project pooled data from 22 robot types across 21 institutions and reported that a model trained on the combined data improved performance on multiple robots. Simulation and synthetic data are also used to fill gaps.
These models are research-grade in most industrial settings. They generalize better to new objects and instructions than older task-specific models, but success rates, cycle times, and safety guarantees are not yet at the level of a programmed industrial cell. They are most promising for high-mix tasks that are hard to program by hand.
Questions about Foundation models for robotics
What is a vision-language-action (VLA) model?
It is one kind of foundation model for robotics. RT-2, from 2023, is a notable example: it fine-tunes a vision-language model on robot data and outputs robot actions as tokens, like words.
Are robot foundation models ready for production lines?
Not in most industrial settings, where they are research-grade. Success rates, cycle times and safety guarantees are not yet at the level of a programmed industrial cell. For a fixed, high-volume task, a conventionally programmed robot remains the reference.
Why is data the bottleneck for robot foundation models?
Robot demonstrations are expensive to collect. The Open X-Embodiment project pooled data from 22 robot types across 21 institutions, and simulation and synthetic data are also used to fill gaps.
Related terms
Physical AI
Physical AI is AI that perceives, reasons about, and acts in the physical world, typically through robots, vehicles, or machines.
Synthetic data
Synthetic data is artificially generated data that mimics some statistical properties of real data.
Bin picking
Bin picking is a robotic task in which a robot, guided by a vision system, locates and grasps individual parts from a container where they lie in random positions and orientations.
Cobot (collaborative robot)
A cobot (collaborative robot) is an industrial robot designed to work in close proximity to people, with safety functions that limit speed, force, or distance so that fencing can be reduced or removed after a risk assessment.
Industrial AI agent
An industrial AI agent is, in loose current usage, a software system that pursues a goal in a plant setting by reading data, reasoning over it, and taking or proposing actions through connected tools, usually built around a large language model.