Brilliant in simulation
A team trains a walking policy for a quadruped in simulation. After a few hours on a GPU it runs, turns, climbs steps and recovers from shoves, thousands of times without a fall. They copy the network onto the real robot, hold it up in a safety harness, lower it to the floor — and it shudders, stamps and topples. Nothing in the network is broken. The simulator was wrong in small ways, and the policy had learned to depend on them.
The Simulation lesson listed the usual culprits: contact and friction models, actuator dynamics and latency, unmodelled flexibility, sensor noise, and, for cameras, how a rendered image differs from a real one. The reinforcement-learning lesson showed why they bite so hard: an optimiser rewarded for performance will find any modelling error that helps and lean on it, just as it found the farmable reward in the corridor.
This lesson is about getting a policy across that gap. There are three complementary tools, and every successful system uses some mix of them:
- System identification: measure the real robot and make the simulator more like it.
- Domain randomisation: train on many deliberately different simulators, so that the real robot is just one more variation.
- Teacher–student training and adaptation: let the policy work out, from its own recent experience, which variation it is in.
You will implement each on a small problem, then see how they were combined in the results that made sim-to-real famous.
The reality gap, precisely
Write for the physical parameters a simulator needs: masses, inertias, friction coefficients, motor constants, delays, sensor noise levels, terrain properties. A simulator with parameters gives an expected return for policy . The real robot behaves like some unknown — or, more honestly, like no value of at all, since every simulator leaves something out.
Definition— Reality gap
For a policy trained in a simulator with parameters , the reality gap is the difference between its performance in simulation and on the real system. A policy transfers zero-shot if it performs acceptably on the real system with no further training there.
The three tools attack different terms. System identification chooses to make the simulator's behaviour match logged real data,
where are measured signals (joint velocities, positions, forces) under the same commands. Domain randomisation replaces the single simulator by a distribution and trains
hoping that the real system looks like a typical draw. Adaptation lets the policy depend on its history of observations and actions, , so that it can behave differently in different : implicitly identifying the system while it acts.
System identification: fitting a joint to its log
Most of the reality gap of a legged robot or an arm sits in its actuators: the motor, the gearbox and the drive electronics. Catalogue values for inertia and friction are a starting point, but gearbox friction depends on lubrication, temperature and wear, and the reflected inertia of the rotor through the gearbox is easy to get wrong.
Consider one joint under torque (current) control. A simple model has rotor-plus-link inertia , viscous friction and Coulomb friction :
Sampled every , this is linear in the unknowns :
so logging and and solving one least-squares problem identifies all three. The program plays the role of the real joint (with a cogging ripple and encoder noise that the model does not include), excites it with a chirp, a sine wave sweeping from 0.2 Hz to about 3 Hz, fits the model, and then compares the catalogue model and the identified one on a different input, a sequence of torque steps.
import numpy as np
# The real joint, which we can only log: inertia J, viscous friction b, Coulomb friction tc,
# plus a small cogging ripple that no one put in the model.
J_TRUE, B_TRUE, TC_TRUE, COG = 0.012, 0.03, 0.15, 0.02
DT = 0.005 # 200 Hz logging
def simulate(tau, J, b, tc, cog=0.0, rng=None, noise=0.0):
th, w, out = 0.0, 0.0, []
for t in tau:
out.append(w + (rng.normal(0, noise) if rng is not None else 0.0)) # what the encoder reports
drive = t - cog * np.sin(8 * th)
if abs(w) < 1e-3 and abs(drive) <= tc: # stuck: static friction holds it
w = 0.0
else:
w = w + DT * (drive - b * w - tc * np.sign(w if abs(w) >= 1e-3 else drive)) / J
th += w * DT
return np.array(out)
rng = np.random.default_rng(0)
time = np.arange(0, 10, DT)
chirp = 0.5 * np.sin(2 * np.pi * (0.2 + 0.14 * time) * time) # 0.2 Hz rising to about 3 Hz
w_log = simulate(chirp, J_TRUE, B_TRUE, TC_TRUE, COG, rng, noise=0.05)
# Equation error: (w[k+1] - w[k]) / dt = (1/J) tau - (b/J) w - (tc/J) sign(w), linear in (1/J, b/J, tc/J).
moving = np.abs(w_log[:-1]) > 0.2
Phi = np.stack([chirp[:-1], -w_log[:-1], -np.sign(w_log[:-1])], axis=1)[moving]
acc = (np.diff(w_log) / DT)[moving]
c = np.linalg.lstsq(Phi, acc, rcond=None)[0]
J_hat, b_hat, tc_hat = 1 / c[0], c[1] / c[0], c[2] / c[0]
print(f"identified: J = {J_hat:.4f} kg m^2 (true {J_TRUE}), b = {b_hat:.4f} N m s (true {B_TRUE}), "
f"tc = {tc_hat:.3f} N m (true {TC_TRUE})")
# Validate on an input the fit never saw: a sequence of torque steps.
steps = np.repeat([0.4, 0.0, -0.3, 0.25, -0.4, 0.0], 400)
w_real = simulate(steps, J_TRUE, B_TRUE, TC_TRUE, COG)
for name, (J, b, tc) in [("datasheet model", (0.009, 0.015, 0.05)), ("identified model", (J_hat, b_hat, tc_hat))]:
pred = simulate(steps, J, b, tc)
print(f"{name:17s}: RMS speed error on held-out steps = {np.sqrt(np.mean((pred - w_real) ** 2)):.2f} rad/s"
f" (real speeds reach {np.abs(w_real).max():.1f} rad/s)")From ten seconds of data the fit recovers (true 0.012), (true 0.03) and (true 0.15). On the held-out torque steps, the identified model predicts the joint's speed to within 0.12 rad/s RMS; the catalogue model, with friction a factor of two to three too low, is off by 8.19 rad/s — as large as the speeds themselves (up to 8.3 rad/s), because it predicts the joint spinning nearly three times too fast.
Four habits of practice are visible here. Excite the system: a chirp exercises a range of speeds and accelerations, while a constant torque would teach you nothing about inertia. Choose which data to believe: the fit skips every sample slower than 0.2 rad/s, where flips with the noise and static friction holds the joint in a way the equation does not describe. Validate on different data: a model that fits its own training log can still be wrong. And know what you left out: the cogging ripple is not in the model, which is why the identified model is good but not perfect, and in a real joint backlash, temperature-dependent friction and current-loop dynamics would add more.
When the physics is too messy for a few parameters, learn it. Jemin Hwangbo and colleagues at ETH Zurich (2019) trained a small neural network on data logged from ANYmal's series-elastic actuators to predict torque from the recent history of position errors and velocities, and put that actuator network inside the simulator; policies trained with it transferred to the real robot where those trained with an idealised motor did not. Learned residual models, which correct a physics simulator using real data, follow the same idea.
Domain randomisation
Identification narrows the gap but never closes it, and some parameters (the friction of every floor a robot will ever walk on, the mass of every box it will lift) cannot be measured in advance. Domain randomisation accepts the uncertainty and trains for all of it. Josh Tobin and colleagues (OpenAI, 2017) randomised the visual appearance of simulated scenes — textures, lighting, camera position — until an object detector trained only on renders worked on real camera images. Xue Bin Peng and colleagues (2018) randomised the dynamics — masses, friction, damping, latency — and transferred a pushing policy to a real arm.
Here is the idea at small scale. A carriage on a rail must move 0.2 m and stop, under a PD policy whose two gains are the "network" to be trained. Commands reach the motor 30 ms late, and the force is limited to 50 N. The engineers' model says the carriage has mass 1.0 kg and viscous friction 20 N s/m — the CAD model includes a heavily damped linear bearing. The real carriage, unknown to them, is 0.6 kg with friction 1.0 N s/m: the bearing was replaced with a smoother one. Both gains are trained with the cross-entropy method, a simple derivative-free optimiser: sample gains, score each in simulation, keep the best fifth, refit. One run trains on the nominal model only; the other on 16 robots whose mass is drawn uniformly from 0.5–2.0 kg and friction from 0.5–20 N s/m.
import numpy as np
# A carriage on a rail must move 0.2 m and stop. The policy is u = kp (0.2 - x) - kd v, computed every
# 10 ms and applied 30 ms late (sensing, computation and the bus), with a 50 N force limit.
# Unknown to the policy: the moving mass m (kg) and the viscous friction b (N s/m).
DT, DELAY, STEPS, UMAX, TARGET = 0.01, 3, 200, 50.0, 0.2
def simulate(kp, kd, m, b):
"""Vectorised over every argument. Returns the cost (time-weighted |error| plus a little effort)
and the largest position reached, for each (gains, robot) pair."""
kp, kd, m, b = np.broadcast_arrays(*map(np.asarray, (kp, kd, m, b)))
x, v, cost, peak = (np.zeros(kp.shape) for _ in range(4))
queue = [np.zeros(kp.shape)] * DELAY
for _ in range(STEPS):
queue.append(np.clip(kp * (TARGET - x) - kd * v, -UMAX, UMAX))
u = queue.pop(0) # the command issued DELAY steps ago
v = v + DT * (u - b * v) / m
x = x + DT * v
cost += DT * (np.abs(TARGET - x) + 0.001 * np.abs(u))
peak = np.maximum(peak, x)
return cost, peak
def train(sample_robots, rng, iters=25, pop=60):
"""Cross-entropy method: sample gains, score each on the training robots, refit to the best fifth."""
mu, sd = np.array([100.0, 10.0]), np.array([60.0, 8.0])
m, b = sample_robots(rng, 16) # the 16 simulated robots used for training
for _ in range(iters):
G = np.abs(mu + sd * rng.standard_normal((pop, 2)))
score = simulate(G[:, :1], G[:, 1:], m[None, :], b[None, :])[0].mean(axis=1)
elite = G[np.argsort(score)[: pop // 5]]
mu, sd = elite.mean(axis=0), elite.std(axis=0) + 0.5
return mu
nominal = lambda rng, n: (np.full(n, 1.0), np.full(n, 20.0)) # the CAD model: 1 kg, heavy damping
randomised = lambda rng, n: (rng.uniform(0.5, 2.0, n), rng.uniform(0.5, 20.0, n))
rng = np.random.default_rng(0)
g_nom, g_dr = train(nominal, rng), train(randomised, rng)
print(f"trained on the nominal model: kp = {g_nom[0]:6.1f}, kd = {g_nom[1]:5.1f}")
print(f"trained with randomisation: kp = {g_dr[0]:6.1f}, kd = {g_dr[1]:5.1f}\n")
for name, (m, b) in [("nominal model m=1.0 b=20", (1.0, 20.0)), ("the real robot m=0.6 b=1.0", (0.6, 1.0)),
("outside the range m=0.3 b=0.2", (0.3, 0.2))]:
(c1, p1), (c2, p2) = simulate(*g_nom, m, b), simulate(*g_dr, m, b)
print(f"{name}: cost {float(c1):.3f} vs {float(c2):.3f} overshoot {100 * max(float(p1) / TARGET - 1, 0):5.1f}% "
f"vs {100 * max(float(p2) / TARGET - 1, 0):5.1f}%")
m_test, b_test = randomised(np.random.default_rng(5), 500) # 500 robots never seen in training
for name, g in [("nominal-trained", g_nom), ("randomised", g_dr)]:
c = simulate(g[0], g[1], m_test, b_test)[0]
print(f"{name:15s} on 500 unseen robots: mean cost {c.mean():.3f}, worst {c.max():.3f}")The policy trained on the nominal model learned to lean on the simulated bearing: its derivative gain is only 8.9, because 20 N s/m of friction did most of the braking for it. In simulation it is excellent (cost 0.026). On the real carriage it overshoots the target by 92% and rings, at almost ten times its cost in simulation. The randomised policy doubles its derivative gain to 18.0, accepts a slightly slower response on the nominal model (cost 0.038 instead of 0.026 — the price of robustness), and on the real carriage overshoots by 20% with a cost of 0.049. Across 500 robots it never saw, its worst case (0.162) is less than half the nominal policy's (0.371).
The third test is a warning. A carriage of 0.3 kg with almost no friction lies outside the randomised range, and both policies overshoot badly (323% and 80%). Randomisation buys robustness over the range you randomise, and a little beyond it — note that none of the 16 training robots had friction as low as the real carriage — but not everywhere. Choosing the ranges is the real engineering: too narrow and reality falls outside; too wide and the policy becomes timid, giving up performance on every robot to survive implausible ones. OpenAI's automatic domain randomisation (2019) widened each range gradually, only as far as the policy could still succeed, while Chebotar and colleagues (2019) went the other way, using a few real trials to adjust the randomisation distribution towards reality.
Teacher, student and privileged information
A randomised policy that cannot tell which robot it is on must use one compromise behaviour for all of them. A better policy would notice: the carriage accelerates more than expected, so it is light; slow down. Two ideas make this practical.
Privileged information. In simulation, everything is known: the true mass and friction, the exact terrain height under each foot, contact forces, the positions of objects behind the camera. A teacher policy given this privileged information is much easier to train with RL, because its problem is fully observed.
Distillation into a student. A student policy receives only what the real robot will have — proprioception, a camera, the recent history of its own commands and sensor readings — and is trained, still in simulation, to reproduce the teacher. The training is the imitation learning of the previous lesson, and because the teacher can be queried in any simulated state, DAgger works perfectly: run the student, label its states with the teacher's actions, retrain. Joonho Lee and colleagues used this to take ANYmal over rubble, mud and streams (2020); RMA (Ashish Kumar and colleagues, UC Berkeley and Carnegie Mellon, 2021) trained an adaptation module to estimate, from the last half-second of joint states and actions, a compact latent code of the environment that the teacher had been given directly, and ran a Unitree A1 on sand, oily surfaces and hiking trails.
The program below builds a minimal version on the carriage. The teacher is told and and chooses its gains to give every robot the same closed-loop behaviour: matching to gives and . The student is not told; its adaptation module looks at the last 15 commands and velocities it has seen, fits to them by least squares, and plugs its estimates into the teacher's formula. The robust policy is the randomised one from above.
import numpy as np
# The same carriage as above: move 0.2 m and stop, 30 ms delay, 50 N limit, unknown mass and friction.
DT, DELAY, STEPS, UMAX, TARGET = 0.01, 3, 200, 50.0, 0.2
W, ZETA = 14.0, 0.8 # desired closed loop: natural frequency 14 rad/s, damping 0.8
DR_GAINS = (210.9, 18.0) # the robust gains found by domain randomisation above
def teacher_gains(m, b):
"""The teacher is told m and b (privileged information, available only in simulation) and places
the closed-loop poles exactly: m s^2 + (b + kd) s + kp = m (s^2 + 2 zeta W s + W^2)."""
return m * W ** 2, max(2 * ZETA * W * m - b, 0.0)
def run(m, b, who, window=15):
x = v = cost = 0.0
queue = [0.0] * DELAY
us, vs = [], []
m_hat, b_hat = 1.0, 5.0 # the student's guess before it has seen anything
for k in range(STEPS):
if who == "teacher":
kp, kd = teacher_gains(m, b)
elif who == "student":
if len(vs) > window: # adaptation module: estimate (m, b) from recent history only
a = np.diff(vs[-window - 1:]) / DT # accelerations it caused
vv, uu = np.array(vs[-window - 1:-1]), np.array(us[-window - 1:-1])
# least squares for u = m a + b v, via the 2x2 normal equations
M = np.array([[a @ a, a @ vv], [a @ vv, vv @ vv]])
if np.linalg.det(M) > 1e-6 * np.trace(M) ** 2: # only when the data is informative
m_hat, b_hat = np.linalg.solve(M, [a @ uu, vv @ uu])
m_hat, b_hat = max(m_hat, 0.2), max(b_hat, 0.0)
kp, kd = teacher_gains(m_hat, b_hat)
else:
kp, kd = DR_GAINS
queue.append(float(np.clip(kp * (TARGET - x) - kd * v, -UMAX, UMAX)))
u = queue.pop(0)
us.append(u); vs.append(v)
v += DT * (u - b * v) / m
x += DT * v
cost += DT * (abs(TARGET - x) + 0.001 * abs(u))
return cost, (m_hat, b_hat)
print("on the real robot (m = 0.6 kg, b = 1.0 N s/m):")
for who in ["teacher", "student", "robust"]:
cost, est = run(0.6, 1.0, who)
extra = f" student's final estimate m = {est[0]:.2f}, b = {est[1]:.2f}" if who == "student" else ""
print(f" {who:8s} cost {cost:.3f}{extra}")
rng = np.random.default_rng(5)
robots = list(zip(rng.uniform(0.5, 2.0, 200), rng.uniform(0.5, 20.0, 200)))
print("\non 200 random robots from the training range:")
for who in ["teacher", "student", "robust"]:
c = np.array([run(m, b, who)[0] for m, b in robots])
print(f" {who:8s} mean cost {c.mean():.3f}, worst {c.max():.3f}")On the real carriage the student's estimates converge to , , and its cost (0.029) is close to the teacher's (0.024) and well below the robust policy's (0.049). Over 200 random robots the student's worst case is 0.038 against the robust policy's 0.158: knowing which robot you are on beats being merely tolerable on all of them.
The robots that crossed the gap
A few results, chosen because their papers are open and their ideas recur, show how the tools combine on real machines.
Legged locomotion (ETH Zurich and others, from 2019). ANYmal's learned controllers began with Hwangbo's actuator network plus dynamics randomisation (2019); Lee's teacher–student policies crossed rough natural terrain blind (2020); Miki and colleagues (2022) added perception, fusing a height map built from depth cameras or lidar with proprioception through a learned module that decides how far to trust the map — enough to finish an alpine hiking route in Switzerland. RMA showed rapid adaptation on a low-cost commercial quadruped from Unitree. As described in Legged locomotion, RL policies trained this way now run on commercial quadrupeds, and on humanoids: Ilija Radosavovic and colleagues (UC Berkeley, 2023) trained a transformer policy for Agility Robotics' Digit entirely in simulation, with heavy randomisation, and deployed it zero-shot outdoors.
In-hand manipulation (OpenAI, 2018–2019). A Shadow Dexterous Hand learned to reorient a block to commanded faces, using a policy trained entirely in a randomised simulator, with vision networks trained on randomised renders to estimate the block's pose. The follow-up solved a Rubik's Cube one-handed, using automatic domain randomisation and a recurrent policy whose memory let it adapt to perturbations it had never seen, including a glove on the hand and objects poking it. NVIDIA's DeXtreme (2022) reproduced cube reorientation on the cheaper Allegro hand with GPU simulation, and Tao Chen and colleagues at MIT (2022) reoriented previously unseen object shapes with a single depth camera, even with the hand facing downwards.
Drone racing (University of Zurich, 2023). The Swift system's policy was trained in simulation with RL, but the team did not trust the simulator blindly: they measured the gap on the real drone and fitted data-driven residual models of the perception and dynamics errors, which were added to the simulator for training. The result raced three human champions and won most of its races against them.
Note
What these projects have in common is less glamorous than the videos: weeks of careful actuator and latency measurement, randomisation ranges chosen and re-chosen, a second simulator used as a sanity check, and a staged route to hardware — first a robot on a stand, then in a harness or gantry that catches falls, then free. Whenever you test a learned policy for the first time, use the same staging on a small, slow robot, with a person at the emergency stop; Embedded control and safety describes the protective layers underneath.
As of 2026 the tooling has become standard. GPU-parallel simulators — NVIDIA's Isaac Lab, Google DeepMind's MuJoCo with its MJX and Playground frameworks, the open-source Genesis project and others — train locomotion policies for quadrupeds and humanoids in hours, and ship with randomisation and teacher–student recipes. The open problems are elsewhere: contact-rich manipulation of soft, articulated or unfamiliar objects, where simulators are least trustworthy; vision, where rendered and real images still differ in subtle ways; and evaluation, since you cannot know a policy has crossed the gap until you have tested it on the real robot many times (next lesson). Building simulations from real scans, so-called real-to-sim, is an active response to all three.
Common misconceptions
Misconception
"A more accurate simulator makes randomisation unnecessary." Identification shrinks the gap but cannot remove it, and many parameters vary in deployment (floors, payloads, wear, temperature). Good systems identify what can be measured and randomise what cannot.
Misconception
"Randomise everything as widely as possible, to be safe." Wider ranges cost performance on every robot, can make the task impossible to learn, and still offer no guarantee outside the ranges. The carriage policy paid 46% extra cost on the nominal model for its robustness, and both policies failed on the robot outside the range.
Misconception
"If it works in simulation across thousands of randomised robots, it will work on the real one." Only if reality is inside what was randomised and the simulator's structure is right. Unmodelled effects — a flexible link, a sensor that saturates, cable drag — are not covered by randomising the parameters of a model that lacks them.
Misconception
"The student is just a smaller copy of the teacher." It solves a harder problem: acting well without the privileged information. Its history-processing module is doing implicit system identification, and a student can never be better than what its observations allow it to infer.
Exercises
1. Crossing the gap from simulation to reality
Warm-upHumanA warehouse robot's wheel motors can be tested on a bench, but it will drive on floors whose friction varies from building to building and from day to day. Which combination is most appropriate before training its controller in simulation?
2. Crossing the gap from simulation to reality
CoreComputationalWrite identify(tau, w, dt) that fits the joint model to logged torques tau and velocities w (NumPy arrays of equal length, sampled every dt seconds) and returns (J, b, tc). Use the equation-error least-squares method from the lesson, and ignore samples where rad/s.
3. Crossing the gap from simulation to reality
Warm-upHumanThe lesson's teacher chooses PD gains so that . For a carriage with kg and N s/m, with rad/s and , what is (in N s/m)?
4. Crossing the gap from simulation to reality
CoreComputationalThe lesson trained gains with the cross-entropy method. Write cem_update(samples, scores, elite_frac) that takes an array of candidate parameter vectors samples (shape (n, d)), their scores (shape (n,), lower is better), and the fraction to keep, and returns the new sampling mean and standard deviation (each shape (d,)) computed from the best int(n * elite_frac) samples.
5. Crossing the gap from simulation to reality
CoreHumanA quadruped's teacher policy is trained in simulation. Which of these inputs are privileged — available to the teacher in simulation but not to the deployed robot? Select all that apply.
6. Crossing the gap from simulation to reality
CoreHumanA policy was trained with payload masses randomised between 0 and 5 kg. A customer asks whether it will work with an 8 kg payload. What is the best answer?
7. Crossing the gap from simulation to reality
Warm-upHumanA robot's controller runs at 200 Hz. Measurements show 25 ms between reading the sensors and the motor applying the resulting command. How many control periods of delay should the simulator's command queue hold to model this?
Further reading and resources
- Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World · Josh Tobin et al. · 2017
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization · Xue Bin Peng et al. · 2018
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots · Jie Tan et al. · 2018Google's Minitaur gaits, transferred using an identified actuator model, latency modelling and randomisation.
- Learning agile and dynamic motor skills for legged robots · Jemin Hwangbo et al. · 2019ANYmal (ETH Zurich) with a neural-network model of its actuators learned from real data.
- Learning quadrupedal locomotion over challenging terrain · Joonho Lee et al. · 2020Teacher–student training with privileged terrain information.
- Learning robust perceptive locomotion for quadrupedal robots in the wild · Takahiro Miki et al. · 2022
- RMA: Rapid Motor Adaptation for Legged Robots · Ashish Kumar, Zipeng Fu, Deepak Pathak and Jitendra Malik · 2021
- Learning Dexterous In-Hand Manipulation · OpenAI · 2018
- Solving Rubik's Cube with a Robot Hand · OpenAI · 2019Automatic domain randomisation.
- DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality · Ankur Handa et al. · 2022
- Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes · Tao Chen et al. · 2022
- Real-World Humanoid Locomotion with Reinforcement Learning · Ilija Radosavovic et al. · 2023
- Champion-level drone racing using deep reinforcement learning · Elia Kaufmann et al. · 2023
- Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience · Yevgen Chebotar et al. · 2019
- MuJoCo Playground · Kevin Zakka et al. · 2025An open-source framework for GPU-parallel training and sim-to-real transfer on legged robots, humanoids, hands and arms.
- Isaac Lab · NVIDIA and contributorsAn open-source framework for robot learning on GPU-parallel simulation.
- System Identification: Theory for the User (2nd edition) · Lennart Ljung · 1999The standard reference on identifying dynamic models from data (Prentice Hall).