No internet of actions
A language model learns from text that people wrote for other reasons: books, articles, code, forums. Trillions of words were already lying around. A robot policy needs something that was never lying around: synchronised recordings of what a robot saw and what it did, on a body like its own, while succeeding at tasks. Every one of those recordings was made by a robot in a room, usually with a person operating it, one episode at a time.
That is why, after the methods of the previous lessons matured, data became the bottleneck of robot learning. Behaviour cloning works (Imitation learning), and simulation helps where physics can be trusted (Sim-to-real). But a policy that can fold laundry in your home, after being trained in someone else's lab, needs experience of many homes, many garments and many robots. Since about 2022, laboratories and companies in North America, Europe and Asia have responded with data collection on a scale robotics had never attempted, and with shared datasets pooled across institutions.
This lesson is a briefing on that effort as of late 2026: how data is collected and what it costs, the main public datasets, how one policy is trained on many different robots, what the evidence on scaling actually shows, how data goes wrong, and why evaluation — knowing whether a policy is any good — is now the hardest part. The models trained on this data are the vision-language-action models of the next level.
Five ways to get robot data
| Source | What you get | Strengths | Weaknesses |
|---|---|---|---|
| Teleoperation | observations + the operator's commands | high-quality actions for exactly this robot | slow, needs skilled people and robots |
| Play | long, unscripted teleoperated sessions | cheap to label, rich in transitions between skills | no task boundaries; needs relabelling |
| Autonomous collection | the robot's own attempts, with automatic success detection | runs unattended, scales with robots | needs resets, safety, and a way to score attempts |
| Human video | people doing tasks, filmed | enormous, diverse, nearly free | no robot actions; human hands are not grippers |
| Simulation and synthesis | rendered observations + exact actions | unlimited, perfectly labelled | the reality gap, and the cost of building scenes |
Teleoperation is the workhorse (see the rigs in Imitation learning). Play data (Corey Lynch and colleagues, Google, 2019) asks operators to do whatever comes to mind in a scene for hours; any stretch of the recording can later be labelled with the goal it happened to reach, so nothing is wasted. Autonomous collection came first historically: Lerrel Pinto and Abhinav Gupta at Carnegie Mellon collected 50,000 grasp attempts over 700 robot hours (2015), and a Google team ran six to fourteen arms at once for two months to collect over 800,000 grasp attempts (2016). Grasping suits it because success is easy to detect (is something in the gripper?) and failure is harmless.
Human video is the largest source by far. Ego4D (2021), recorded by a consortium of universities across nine countries, contains 3,670 hours of first-person video of daily life. It contains no robot actions, but it can teach visual representations (R3M, 2022, pretrained on Ego4D, improved downstream robot imitation), and hand-tracking can turn some human motions into robot targets — a line of work that grows each year. Handheld collection devices such as UMI (Imitation learning) sit between the two: humans do the task, but with a gripper and camera the robot will later carry. Synthetic data multiplies what humans provide: MimicGen (NVIDIA and UT Austin, 2023) generated over 50,000 demonstrations across 18 tasks from about 200 human ones, by transforming each human demonstration to new object positions in simulation and keeping only the variants that succeeded.
What a demonstration costs
Budgets for data collection are decided by arithmetic like the program below. Every number in it is an assumption, chosen to be plausible for a tabletop arm task rather than taken from any particular project; change them to match your own situation. Two costs are often forgotten: the time to reset the scene between attempts, and the time to move a station to a new scene and recalibrate its cameras, which is what diversity costs.
# What does it cost to collect teleoperated demonstrations? Every number below is an ASSUMPTION
# for illustration: change them to match your own robot, task and country.
ATTEMPT_S = 45 # seconds of teleoperation per attempt
RESET_S = 25 # seconds to reset the scene between attempts
KEEP = 0.85 # fraction of attempts that succeed and are kept
PRODUCTIVE = 0.70 # fraction of a shift spent recording (breaks, calibration, faults)
SCENE_SETUP_H = 2.0 # hours to move a station to a new scene and recalibrate its cameras
OPERATOR_USD_H = 25.0 # per operator hour, fully loaded (varies widely between countries)
ENGINEER_USD_H = 70.0 # per hour of engineering support (repairs, software, data pipeline)
ENGINEER_RATIO = 0.2 # engineering hours needed per operator hour
STATION_USD = 40_000 # one station: arms, cameras, computer, teleoperation rig
STATION_LIFE_H = 8_000 # hours of use before the station is worn out or obsolete
def collect(demos, scenes):
"""Returns robot-station hours and total cost in USD for `demos` kept demonstrations in `scenes` scenes."""
recording_h = demos / KEEP * (ATTEMPT_S + RESET_S) / 3600 / PRODUCTIVE
station_h = recording_h + scenes * SCENE_SETUP_H
operator_h = station_h
cost = (operator_h * (OPERATOR_USD_H + ENGINEER_RATIO * ENGINEER_USD_H)
+ station_h * STATION_USD / STATION_LIFE_H)
return station_h, cost
print(f"{'demos':>9} {'scenes':>7} {'station hours':>14} {'cost (USD)':>12} {'USD per demo':>13}")
for demos, scenes in [(200, 1), (10_000, 1), (10_000, 200), (76_000, 564), (1_000_000, 5_000)]:
hours, cost = collect(demos, scenes)
print(f"{demos:>9,} {scenes:>7,} {hours:>14,.0f} {cost:>12,.0f} {cost / demos:>13.2f}")
hours, _ = collect(1_000_000, 5_000)
print(f"\n1,000,000 demos = {hours / (16 * 250):.0f} station-years at two shifts a day, 250 days a year")
print(f"share of recording time the robot actually moves under teleoperation: "
f"{ATTEMPT_S / (ATTEMPT_S + RESET_S) * PRODUCTIVE:.0%}")Under these assumptions a demonstration costs about USD 1.50 to 3.20, and a million of them about USD 1.9 million and eleven station-years — affordable for a large organisation, out of reach for most universities. Three things stand out.
- Diversity is what costs. Ten thousand demonstrations in one scene take 329 station hours; spread over 200 scenes they take 727, more than twice as long, because every move means unpacking, setting up and recalibrating. The 76,000-demonstration row uses DROID's published scale — 76,000 trajectories in 564 scenes — as a reference point.
- Most station time is not data. Only 45% of recording time is the robot moving under teleoperation; resets, breaks and faults take the rest, before counting scene changes.
- The real bill is higher. The model leaves out management, the people who design tasks and check data, robot repairs, failed hardware, storage and the long tail of practical problems. Large real campaigns report durations of a year or more: DROID's 76,000 trajectories were collected by 50 people over 12 months, and RT-1's 130,000 episodes over 17 months with 13 robots.
The shared datasets
Before 2023, most robot-learning papers trained on data their own lab collected for that paper. Pooling began with RoboNet (2019: 15 million video frames from 7 robot platforms in several labs) and grew rapidly.
- RT-1 (Google, 2022): over 130,000 episodes covering more than 700 tasks, collected with a fleet of 13 mobile manipulators in offices and kitchens over 17 months.
- BridgeData V2 (UC Berkeley and collaborators, 2023): 60,096 trajectories in 24 environments on a low-cost WidowX arm, a robot many labs can afford, which made it a common benchmark.
- RH20T (Shanghai Jiao Tong University, 2023): over 110,000 contact-rich sequences recorded with force–torque sensing as well as cameras, across several robot configurations.
- Open X-Embodiment (2023): a collaboration of 21 institutions that pooled 60 existing datasets from 34 labs into over a million trajectories from 22 robot embodiments, in a common format, and trained the RT-X models on them.
- DROID (2024): 76,000 trajectories, 350 hours, collected "in the wild" with identical Franka arm setups in 564 scenes and 84 tasks by 50 collectors in North America, Asia and Europe over 12 months — designed for diversity of scenes rather than repetition.
- AgiBot World (AgiBot, Shanghai, 2025): over a million trajectories across 217 tasks in five deployment scenarios, collected on the company's own robots in a dedicated data-collection facility.
Community datasets in a shared format are also published every week on the LeRobot hub. Several companies (for example Physical Intelligence, Google DeepMind, Tesla and Figure, as of 2026) train on large internal datasets that are not released, so their size and content cannot be checked independently.
One policy, many robots
Pooling data from 22 kinds of robot raises an obvious problem: their bodies differ. Arms have six or seven joints, grippers are parallel jaws or hands, cameras sit in different places, and controllers run at 3 Hz on one robot and 50 Hz on another. A cross-embodiment policy must accept all of these. There are three main strategies.
- A common action space. Express every dataset's actions in the same form — commonly the change in end-effector position and orientation plus a gripper command — and normalise each dataset's statistics. The RT-X models took this route. It is simple but discards whatever does not fit the common form (a hand's finger joints, a mobile base).
- Embodiment-specific inputs and outputs around a shared trunk. Small per-robot encoders ("stems") turn each robot's proprioception and camera images into tokens; a large transformer trunk is shared; per-robot heads decode actions. HPT (MIT and Meta, 2024) pre-trained such a trunk on 52 datasets, including simulation and human video.
- Tokenised actions. Discretise or compress action sequences into tokens and let a single sequence model predict them, so different embodiments are just different token streams; this is how many vision-language-action models handle it.
Does pooling help? The evidence so far says: often, and most for the robots with the least data of their own. In the Open X-Embodiment study, RT-1-X, trained on the pooled data, outperformed models trained on each dataset alone by 50% on average in the small-data domains, and the larger RT-2-X performed three times better than RT-2 on "emergent skills" that appeared only in other robots' data. Octo (UC Berkeley, Stanford, Carnegie Mellon and others, 2024) trained an open-source generalist policy on 800,000 Open X-Embodiment trajectories, designed to be fine-tuned to new robots with modest data. But transfer is uneven: robots very unlike the bulk of the data benefit less, and badly mixed data can hurt. How much of each dataset to sample is itself a research question; Re-Mix (Stanford and UC Berkeley, 2024) learned mixture weights that beat uniform weighting by 38% on average.
What the scaling evidence shows
In language modelling, loss falls as a smooth power law in model size, data and compute, and labs use those laws to plan training runs (Language models). Does robotics have anything comparable? As of 2026, partly.
The clearest controlled study is by Fanqi Lin and colleagues at Tsinghua University (2024). They collected over 40,000 demonstrations, ran more than 15,000 real-robot evaluation rollouts, and measured how well policies generalised to new environments and new objects. Generalisation followed roughly a power law in the number of distinct environments and objects in the training data. Once each environment had enough demonstrations, more demonstrations of the same environment added little: in their words, diversity of environments and objects mattered far more than the absolute number of demonstrations. In practice, four people collecting for one afternoon produced policies with about 90% success in unseen environments with unseen objects, on the two tasks studied.
The program below is a caricature of why that should be — not evidence, but a way to see the mechanism. Each scene has a hidden property (its lighting, table, objects) that changes the right action; a nearest-neighbour imitator is tested in scenes it has never seen. With a fixed budget of 512 demonstrations, it compares spreading them over few or many scenes, then piles more demonstrations into eight scenes.
import numpy as np
# A caricature of generalisation. Each scene has a hidden property e (lighting, table height, object
# set...) in [0, 1]; the right action depends on the robot's state s AND on e. A nearest-neighbour
# policy imitates demonstrations; it is tested in 1,000 situations from scenes it has never seen.
def expert(s, e):
return np.sin(3 * s + 4 * e) + 0.5 * np.cos(5 * e)
def policy_error(n_scenes, demos_per_scene, rng, k=5):
e_train = np.repeat(rng.uniform(0, 1, n_scenes), demos_per_scene)
s_train = rng.uniform(-1, 1, e_train.size)
a_train = expert(s_train, e_train)
s_test, e_test = rng.uniform(-1, 1, 1000), rng.uniform(0, 1, 1000) # new scenes
d2 = (s_test[:, None] - s_train[None, :]) ** 2 + (e_test[:, None] - e_train[None, :]) ** 2
nearest = np.argpartition(d2, k, axis=1)[:, :k] # the k most similar demos
pred = a_train[nearest].mean(axis=1)
return np.mean(np.abs(pred - expert(s_test, e_test)) > 0.2) # failure: off by > 0.2
rng = np.random.default_rng(0)
print("same budget of 512 demonstrations, spread over more and more scenes:")
for scenes in [1, 2, 4, 8, 16, 32, 64, 128, 512]:
fails = np.mean([policy_error(scenes, 512 // scenes, rng) for _ in range(5)])
print(f" {scenes:3d} scene{'s' if scenes > 1 else ' '} x {512 // scenes:3d} demos: failure rate {fails:.3f}")
print("\n8 scenes, more and more demonstrations in each:")
for per in [8, 32, 128, 512]:
fails = np.mean([policy_error(8, per, rng) for _ in range(5)])
print(f" 8 scenes x {per:3d} demos ({8 * per:4d} total): failure rate {fails:.3f}")Spread over one scene, 512 demonstrations leave the imitator failing 74% of the time in new scenes; over 64 scenes, the same 512 bring failures down to about 4%. With only eight scenes, going from 256 to 4,096 demonstrations leaves the failure rate stuck between 25% and 35%: the policy has learned those eight scenes thoroughly and still has to guess about the space between them. Compare the cost program: diversity is exactly the expensive part.
What is still open, as of 2026:
- No agreed scaling law. Results like Lin and colleagues' are for specific tasks and policy classes; nobody yet knows the robotics analogue of a compute-optimal recipe, partly because measuring a point on the curve requires hundreds of real trials (next section).
- Can other data substitute? Human video, simulation and synthetic augmentation clearly help, for representations and for some skills, but how many robot hours an hour of human video is worth remains unsettled.
- Contact and long horizons. Most large datasets are dominated by short pick-and-place episodes recorded without force sensing; data on forceful, dexterous or many-minute tasks is scarce.
- Does scale buy reliability? Broad data clearly buys breadth — more tasks and scenes at moderate success. Whether it also buys the 99.9% reliability a deployed product needs, or whether that comes from task-specific data and fine-tuning, is debated (Evaluation, safety and deployment).
What makes data good
Quantity aside, practitioners converge on a checklist. Consistent operators: one strategy per task where possible, few hesitations, and recorded operator identity so that bad sessions can be found. Labels: success or failure for every episode (failures are useful for RL and for evaluators, but must not be cloned), a language description of the task, and the scene and object identities. Synchronisation and calibration: camera and joint timestamps aligned to a few milliseconds, camera intrinsics and extrinsics stored with the data. Documented conventions for frames, units and action spaces. Curation: removing duplicates, broken recordings and idle stretches, and weighting sources sensibly. And consent and privacy, since data recorded in homes and workplaces captures people who did not sign up to be training data.
Evaluation is the hard part
A language model can be scored on thousands of benchmark questions in minutes. A robot policy is scored by running it on a real robot, resetting the scene, and running it again — a few minutes per trial. The scene is never quite the same twice, lighting changes through the day, batteries drain, objects wear, and a policy that worked on Tuesday may fail on Friday for reasons that have nothing to do with the policy. So robot-learning papers usually report success rates from 10 to 50 trials per condition. How much can that tell you?
from math import comb, sqrt
def wilson(k, n, z=1.96):
"""95% Wilson score interval for a success rate after k successes in n trials."""
p = k / n
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
return centre - half, centre + half
print("A policy that succeeds 70% of the time, measured with n trials:")
for n in [10, 20, 50, 100, 200, 500]:
lo, hi = wilson(round(0.7 * n), n)
print(f" {round(0.7 * n):3d}/{n:<3d} 95% interval {lo:.2f} to {hi:.2f} width {hi - lo:.2f}")
def binom(n, p):
return [comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(n + 1)]
def wrong_call(n, p_good=0.8, p_bad=0.7):
"""Exact probability that the worse policy scores higher than, or ties with, the better one."""
good, bad = binom(n, p_good), binom(n, p_bad)
worse = sum(good[i] * bad[j] for i in range(n + 1) for j in range(n + 1) if j > i)
tie = sum(good[i] * bad[i] for i in range(n + 1))
return worse, tie
print("\nComparing a 70% policy with an 80% one, n trials each:")
for n in [10, 20, 50, 100, 200, 300]:
worse, tie = wrong_call(n)
print(f" n = {n:3d}: the worse policy wins {worse:5.1%}, ties {tie:5.1%}")
# Trials per policy for a two-sided test at the 5% level to detect 0.70 vs 0.80 with 80% power.
p1, p2, z_a, z_b = 0.7, 0.8, 1.96, 0.8416
pbar = (p1 + p2) / 2
n = (z_a * sqrt(2 * pbar * (1 - pbar)) + z_b * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
print(f"\ntrials per policy to detect 70% vs 80% reliably: about {n:.0f}")
print(f"at 3 minutes per real trial including reset: {2 * n * 3 / 60:.0f} robot-hours for one comparison")Seven successes in ten trials is consistent with a true success rate anywhere from 40% to 89%. The interval narrows only as : fifty times more trials (500) shrink it six- to seven-fold, to points. Comparing two policies is worse. With 20 trials each, a policy that is truly 10 points worse beats the better one 17.8% of the time and ties 11.2% of the time, so a paper's "ours wins 15/20 to 13/20" is close to a coin toss. Detecting a 10-point difference reliably takes about 293 trials per policy, some 29 robot-hours for a single comparison under the program's assumptions. Lumenary CS's Evaluating models treats the same statistics for language-model benchmarks; robotics simply has far fewer samples to work with.
Definition— Wilson score interval
For successes in independent trials, with and for 95% confidence, the Wilson interval is
Unlike the textbook interval , it stays inside and behaves sensibly for small and for success rates near 0 or 1, which is where robot results usually live.
The community's response, argued for in the best-practice paper by Hadas Kress-Gazit and colleagues (2024) and increasingly common in practice: report the number of trials and confidence intervals, not bare percentages; interleave trials of the policies being compared (A, B, A, B…) from matched initial conditions, so that drifting lighting or wear affects both equally; blind the person scoring outcomes to which policy ran; define success precisely before the experiment, and report partial progress and a taxonomy of failures, not only the success rate. Simulated evaluation suites such as LIBERO and SIMPLER (2024) offer thousands of cheap, reproducible trials, and SIMPLER was built specifically so that its rankings track real-robot results; but a simulator ranks policies only as well as its own reality gap allows. Some groups now also run shared real-robot evaluations that are open to policies from other labs, so that different methods are tested on the same robots under the same conditions. The last lesson of the course returns to evaluation for deployment, where the question is no longer "70% or 80%?" but "how sure are we that the failure rate is below one in ten thousand?"
Common misconceptions
Misconception
"Robot learning just needs more data, like language models did." More of the same data helps little once a task's situations are covered; diversity of scenes, objects and embodiments matters more, and is what makes data expensive. Data quality and conventions matter as much as quantity.
Misconception
"A dataset of a million trajectories is a million times one demonstration's worth." Episodes vary in length and difficulty, pooled datasets overlap, and much of a large dataset may be near-duplicates of common pick-and-place motions. Count distinct scenes, objects, tasks and robots, not only episodes.
Misconception
"Policy A succeeded 16/20 times and policy B 13/20, so A is better." Those results are entirely consistent with the two being equally good; with 20 trials each, even a 10-point real difference is reversed or tied almost 30% of the time. Report intervals, and run enough interleaved trials to justify the claim.
Misconception
"Human video will soon replace robot data." It is already valuable for perception and for learning what to do; it does not show the forces, the robot's own body or the exact actions, so it complements robot data rather than replacing it, at least as of 2026.
Exercises
1. Robot data at scale
Warm-upHumanEach teleoperated attempt takes 40 s, followed by a 20 s reset. 80% of attempts succeed and are kept. How many kept demonstrations does one station produce per hour of uninterrupted recording?
2. Robot data at scale
CoreComputationalWrite wilson(k, n, z=1.96) returning the Wilson score interval (low, high) for successes in trials:
3. Robot data at scale
CoreHumanYou expect a policy's success rate to be about 70% and want a 95% interval of about percentage points. Using the normal approximation, half-width , what is the smallest number of trials that achieves a half-width of at most 0.05?
4. Robot data at scale
CoreComputationalWrite prob_worse_wins(n, p_good, p_bad) returning the exact probability that a policy with true success rate p_bad gets strictly more successes than one with p_good, when each is run n independent times. (Use math.comb for binomial probabilities.)
5. Robot data at scale
CoreHumanYou can afford 1,000 teleoperated demonstrations of a table-clearing task, and the policy must work in kitchens it has never seen. Based on the evidence in the lesson, which plan is most likely to generalise best?
6. Robot data at scale
CoreHumanYou are merging datasets from three labs that used different arms to train one policy with a common end-effector action space. Which of these must you check and, if needed, convert? Select all that apply.
7. Robot data at scale
Warm-upHumanWhy can't thousands of hours of first-person human video simply replace teleoperated robot data for training a manipulation policy?
Further reading and resources
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models · Open X-Embodiment Collaboration · 2023Over a million trajectories from 22 robot embodiments, pooled from 60 datasets; project page at robotics-transformer-x.github.io.
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset · Alexander Khazatsky et al. · 2024
- BridgeData V2: A Dataset for Robot Learning at Scale · Homer Walke et al. · 2023
- RT-1: Robotics Transformer for Real-World Control at Scale · Anthony Brohan et al. · 2022
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot · Hao-Shu Fang et al. · 2023
- AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems · AgiBot World Contributors · 2025
- RoboNet: Large-Scale Multi-Robot Learning · Sudeep Dasari et al. · 2019
- Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours · Lerrel Pinto and Abhinav Gupta · 2015
- Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection · Sergey Levine et al. · 2016
- Learning Latent Plans from Play · Corey Lynch et al. · 2019
- Ego4D: Around the World in 3,000 Hours of Egocentric Video · Kristen Grauman et al. · 2021
- R3M: A Universal Visual Representation for Robot Manipulation · Suraj Nair et al. · 2022
- MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations · Ajay Mandlekar et al. · 2023
- Octo: An Open-Source Generalist Robot Policy · Octo Model Team · 2024
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers · Lirui Wang et al. · 2024
- Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning · Joey Hejna et al. · 2024
- Data Scaling Laws in Imitation Learning for Robotic Manipulation · Fanqi Lin et al. · 2024
- Robot Learning as an Empirical Science: Best Practices for Policy Evaluation · Hadas Kress-Gazit et al. · 2024
- Evaluating Real-World Robot Manipulation Policies in Simulation · Xuanlin Li et al. · 2024SIMPLER, simulated evaluation environments that track real-robot results.
- LeRobot · Hugging FaceOpen-source datasets, data format and tools for sharing robot data.