Skip to content
LumenaryAcademy
Learning 80 min

Reinforcement learning for robot control

Learning a controller from reward and experience — Markov decision processes, Q-learning, policy gradients, PPO and SAC, the art of writing rewards, and why robots do their practising in simulation.

Code tested · 6AI-draftedAwaiting review by a roboticist

Your guides: Chloe · The Builder, Margaret · The Experimenter, Gideon · The Safety Engineer

MotivationH

A controller nobody wrote

Everything you have built so far in this course started from a model. You wrote down the dynamics of a joint and tuned a PID loop for it; you linearised a cart-pole and solved a Riccati equation for its LQR gain; you optimised trajectories through a model. When the model is good, these methods are hard to beat: they are fast, predictable and come with guarantees.

Some tasks resist this. A quadruped crossing loose gravel touches the ground with four feet at unpredictable moments; a hand turning a cube between its fingers makes and breaks a dozen contacts a second. Writing an accurate model of such a task is hard, and designing a controller from it is harder still. Yet it is easy to score an attempt: did the robot move forward without falling? Is the cube in the target orientation?

Reinforcement learning (RL) turns that score into a controller. The robot tries something, receives a number called a reward, and adjusts its behaviour to collect more reward in future. Nobody writes the control law; it is found by trial and error. Since about 2019 this has produced the walking controllers on many research and commercial quadrupeds (see Legged locomotion), a drone that beat human champions at racing (University of Zurich, 2023), and in-hand manipulation that hand-designed controllers had never achieved.

It also has a price: RL needs enormous amounts of experience, exploring can break things, and a sensible-looking reward can be satisfied in ways you never intended. You will implement the two families every modern algorithm builds on — Q-learning and policy gradients — and see why robot RL is done in a simulator first. For gradient descent, networks and training practice, see Lumenary CS: Learning from data, Neural networks and Training in practice.

DefinitionsH

Markov decision processes

RL formalises a robot task as a loop between an agent (the controller being learned) and an environment (the robot's body and everything around it).

policy π(a | s)the controller being learnedrobot + worlddynamics P(s′ | s, a), reward ractionaₜstate sₜ₊₁reward rₜ₊₁
The reinforcement-learning loop. At each step the policy sees the state and picks an action; the robot and the world respond with a new state and a reward. Learning means changing the policy so that the rewards add up to more.

Definition— Markov decision process (MDP)

An MDP is a tuple (S,A,P,r,γ)(\mathcal{S}, \mathcal{A}, P, r, \gamma): a set of states S\mathcal{S}, a set of actions A\mathcal{A}, transition probabilities P(s′∣s,a)P(s' \mid s, a), a reward function r(s,a,s′)r(s, a, s'), and a discount factor 0≤γ<10 \le \gamma \lt 1. It is Markov because the next state depends only on the current state and action, not on how the robot got there.

A policy π(a∣s)\pi(a \mid s) gives the probability of each action in each state (a deterministic policy is the special case a=π(s)a = \pi(s)). Starting from s0s_0 and following π\pi produces a trajectory s0,a0,r1,s1,a1,r2,…s_0, a_0, r_1, s_1, a_1, r_2, \ldots. Its return from time tt is the discounted sum

Gt=rt+1+γrt+2+γ2rt+3+⋯=∑k=0∞γkrt+k+1.G_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}.

The state-value function and the action-value function of π\pi are the expected returns

Vπ(s)=Eπ[Gt∣st=s],Qπ(s,a)=Eπ[Gt∣st=s,at=a].V^\pi(s) = \mathbb{E}_\pi[G_t \mid s_t = s], \qquad Q^\pi(s, a) = \mathbb{E}_\pi[G_t \mid s_t = s, a_t = a].

The goal is a policy π∗\pi^* whose value is as large as possible in every state; its value functions are written V∗V^* and Q∗Q^*.

The discount keeps infinite sums finite and sets a horizon: with γ=0.99\gamma = 0.99 a reward 100 steps away counts for 0.99100≈0.370.99^{100} \approx 0.37 of an immediate one, so the robot plans roughly 1/(1−γ)=1001/(1-\gamma) = 100 steps ahead. At a 50 Hz control rate that is two seconds, about right for a stride or a grasp.

Value functions satisfy a consistency condition, which every algorithm in this lesson exploits. The return is the first reward plus the discounted return from the next state, Gt=rt+1+γGt+1G_t = r_{t+1} + \gamma G_{t+1}, so taking expectations gives the Bellman equations

Qπ(s,a)=Es′[r+γ Ea′∼πQπ(s′,a′)],Q∗(s,a)=Es′[r+γmax⁡a′Q∗(s′,a′)].Q^\pi(s, a) = \mathbb{E}_{s'}\big[r + \gamma \, \mathbb{E}_{a' \sim \pi} Q^\pi(s', a')\big], \qquad Q^*(s, a) = \mathbb{E}_{s'}\big[r + \gamma \max_{a'} Q^*(s', a')\big].

The second, the Bellman optimality equation, says: the value of doing aa now is what you get immediately plus the value of acting optimally afterwards. Given Q∗Q^*, the optimal policy is simply to pick arg⁡max⁡aQ∗(s,a)\arg\max_a Q^*(s, a) in every state.

ComputationC

Q-learning on a warehouse floor

If we knew PP and rr we could find Q∗Q^* by value iteration: start from any QQ, and repeatedly replace it with the right-hand side of the Bellman optimality equation. The update is a contraction (each sweep shrinks the largest error by a factor γ\gamma), so it converges.

Q-learning (Christopher Watkins, 1989) does the same without the model. Each time the robot takes action aa in state ss and observes reward rr and next state s′s', it nudges its estimate towards a one-sample version of the Bellman target:

Q(s,a)←Q(s,a)+α[r+γmax⁡a′Q(s′,a′)⏟TD target−Q(s,a)].Q(s, a) \leftarrow Q(s, a) + \alpha \big[\underbrace{r + \gamma \max_{a'} Q(s', a')}_{\text{TD target}} - Q(s, a)\big].

The bracket is the temporal-difference (TD) error: how much better or worse things turned out than predicted. With a small learning rate α\alpha the noise of individual samples averages out. To explore, the robot usually acts greedily but picks a random action with probability ε\varepsilon (ε\varepsilon-greedy). Q-learning is off-policy: it learns the value of the greedy policy even while behaving partly at random, because the target uses max⁡a′\max_{a'} rather than the action actually taken next.

Our robot is a small delivery cart in a warehouse aisle. It starts at S and must reach the charging dock G. Shelving (#) blocks the row above the aisle; drain grates (X) line the row below, and a wheel that drops into a grate ends the episode (someone has to come and lift the cart out). Each move costs −1-1, reaching the dock pays +10+10, a grate costs −20-20. The floor is slippery: with probability 0.2 the cart slides sideways instead of going where it was told.

shelvingXXXXSGX = drain grate (episode ends, −20)no slip:shortest route20% slip:long way round
The warehouse aisle. The dashed route is the shortest, and optimal when the floor does not slip. With 20% sideways slip, every eastward step along the aisle risks sliding into a grate, and the optimal route (solid) goes the long way round above the shelving.

The program solves the problem twice: once by value iteration, which is allowed to read the model, and once by Q-learning, which only gets to drive the cart and see what happens. To judge Q-learning fairly, it also computes the true expected return of each learned policy exactly, by solving the linear Bellman equations V=R+γPVV = R + \gamma P V for that policy.

python
import numpy as np

# A warehouse aisle seen from above: S start, G charging dock, # shelving, X drain grates.
MAP = ["......",
       ".####.",
       "S....G",
       ".XXXX.",
       "......"]
MOVES, ARROWS = [(-1, 0), (0, 1), (1, 0), (0, -1)], "^>v<"     # north, east, south, west
GAMMA = 0.95
cells = [(r, c) for r in range(5) for c in range(6) if MAP[r][c] != "#"]
index = {cell: i for i, cell in enumerate(cells)}
START = index[(2, 0)]
TERMINAL = {i for i, (r, c) in enumerate(cells) if MAP[r][c] in "GX"}

def outcomes(s, a, slip):
    """(probability, next state, reward) for action a in state s. The wheels slip sideways with probability slip."""
    out = []
    for p, a2 in [(1 - slip, a), (slip / 2, (a + 1) % 4), (slip / 2, (a + 3) % 4)]:
        r, c = cells[s][0] + MOVES[a2][0], cells[s][1] + MOVES[a2][1]
        s2 = index.get((r, c), s)                      # a wall or shelf: stay where you are
        reward = -1.0 + {"G": 10.0, "X": -20.0}.get(MAP[cells[s2][0]][cells[s2][1]], 0.0)
        out.append((p, s2, reward))
    return out

def value_iteration(slip):
    """The model-based answer: iterate the Bellman optimality equation until Q stops changing."""
    Q = np.zeros((len(cells), 4))
    for _ in range(500):
        V = np.where([s in TERMINAL for s in range(len(cells))], 0.0, Q.max(axis=1))
        Q = np.array([[sum(p * (r + GAMMA * V[s2]) for p, s2, r in outcomes(s, a, slip)) for a in range(4)]
                      for s in range(len(cells))])
    return Q

def q_learning(episodes, slip, alpha=0.1, eps=0.2, seed=0):
    """The model-free answer: learn Q from experience alone, one transition at a time."""
    rng = np.random.default_rng(seed)
    Q = np.zeros((len(cells), 4))
    for _ in range(episodes):
        s = START
        for _ in range(200):
            a = int(rng.integers(4)) if rng.random() < eps else int(np.argmax(Q[s]))   # epsilon-greedy
            outs = outcomes(s, a, slip)
            _, s2, r = outs[rng.choice(3, p=[o[0] for o in outs])]                       # the world decides
            target = r if s2 in TERMINAL else r + GAMMA * Q[s2].max()                    # TD target
            Q[s, a] += alpha * (target - Q[s, a])
            s = s2
            if s in TERMINAL:
                break
    return Q

def true_value(policy, slip):
    """Exact expected return of a fixed policy: solve the linear Bellman equations V = R + gamma P V."""
    n = len(cells)
    P, R = np.zeros((n, n)), np.zeros(n)
    for s in set(range(n)) - TERMINAL:
        for p, s2, r in outcomes(s, policy[s], slip):
            P[s, s2] += p
            R[s] += p * r
    return np.linalg.solve(np.eye(n) - GAMMA * P, R)

def show(Q, title):
    print(title)
    for r in range(5):
        print("   " + " ".join(MAP[r][c] if MAP[r][c] in "#GX" else ARROWS[int(np.argmax(Q[index[(r, c)]]))]
                              for c in range(6)))

show(value_iteration(slip=0.0), "optimal policy, no slip:")
Qstar = value_iteration(slip=0.2)
show(Qstar, "optimal policy, 20% sideways slip:")
print(f"optimal value of the start cell: {Qstar[START].max():.2f}\n")
for episodes in [50, 200, 1000, 5000]:
    Q = q_learning(episodes, slip=0.2)
    a = int(np.argmax(Q[START]))
    V = true_value(Q.argmax(axis=1), slip=0.2)
    print(f"Q-learning, {episodes:4d} episodes: first move {ARROWS[a]}, its estimate {Q[START, a]:6.2f}, "
          f"true value of its policy {V[START]:6.2f}")
show(Q, "\npolicy learned by Q-learning (5000 episodes):")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Read the output carefully; it contains most of the lessons of RL in miniature.

  • The optimal policy depends on the noise. Without slip, the best first move from S is east, straight along the aisle. With 20% slip the best move is north: four eastward steps beside the grates would each risk a 10% slide into one, and the detour of four extra steps is cheaper. No one told the robot about risk; the expected-return objective produced it.
  • Estimates can be confidently wrong. After 50 episodes Q-learning believes its first move is worth −3.41-3.41, close to the true optimum of −3.27-3.27, but the policy it would actually follow is worth −19.91-19.91: it heads south and down a route that ends in a grate far too often. A value estimate is only as good as the experience behind it.
  • It gets there. From 200 episodes on, the learned policy starts by going north, and after 5,000 episodes its true value is the optimal −3.27-3.27.
  • Only visited states are learned. In the bottom row, which a good policy never visits, Q-learning's arrows are still wrong after 5,000 episodes. That is harmless here and alarming on a real robot: a learned policy may do something arbitrary in a state it rarely saw during training.
DefinitionsH

Policy gradients

Robot actions are continuous: torques, velocities, joint targets. Taking max⁡a′Q(s′,a′)\max_{a'} Q(s', a') over a continuous action space is itself an optimisation problem, so for control it is often simpler to parametrise the policy directly, as πθ(a∣s)\pi_\theta(a \mid s), and climb the gradient of the expected return

J(θ)=Eτ∼πθ[∑t=0T−1γtrt+1],J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^{T-1} \gamma^t r_{t+1}\Big],

where τ\tau is a whole trajectory. The difficulty is that θ\theta changes which trajectories happen, through dynamics we cannot differentiate. A short identity gets around it.

Theorem— Policy gradient (REINFORCE)

For a policy πθ\pi_\theta that is differentiable in θ\theta, and any baseline b(st)b(s_t) that does not depend on the action,

∇θJ(θ)=Eτ∼πθ[∑t=0T−1∇θlog⁡πθ(at∣st) (Gt−b(st))],\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\Big[\sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \big(G_t - b(s_t)\big)\Big],

where GtG_t is the (discounted) reward collected from time tt onwards.

Proof

Write pθ(τ)p_\theta(\tau) for the probability of a trajectory and R(τ)R(\tau) for its total reward. Since ∇p=p ∇log⁡p\nabla p = p \, \nabla \log p,

∇θJ=∇θ∫pθ(τ)R(τ) dτ=∫pθ(τ) ∇θlog⁡pθ(τ) R(τ) dτ.\nabla_\theta J = \nabla_\theta \int p_\theta(\tau) R(\tau)\, d\tau = \int p_\theta(\tau)\, \nabla_\theta \log p_\theta(\tau)\, R(\tau)\, d\tau .

Now pθ(τ)=p(s0)∏tπθ(at∣st)P(st+1∣st,at)p_\theta(\tau) = p(s_0) \prod_t \pi_\theta(a_t \mid s_t) P(s_{t+1} \mid s_t, a_t), and the dynamics do not depend on θ\theta, so ∇θlog⁡pθ(τ)=∑t∇θlog⁡πθ(at∣st)\nabla_\theta \log p_\theta(\tau) = \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t): the unknown model drops out. Rewards received before time tt cannot be influenced by ata_t, and their products with the score ∇log⁡πθ(at∣st)\nabla \log \pi_\theta(a_t \mid s_t) average to zero, which leaves only the reward-to-go GtG_t. Finally, Ea∼π[∇θlog⁡πθ(a∣s)]=∇θ∫πθ(a∣s) da=∇θ1=0\mathbb{E}_{a \sim \pi}[\nabla_\theta \log \pi_\theta(a \mid s)] = \nabla_\theta \int \pi_\theta(a \mid s)\, da = \nabla_\theta 1 = 0, so subtracting any action-independent b(st)b(s_t) leaves the expectation unchanged.

∎

The estimator says: make actions that were followed by better-than-baseline returns more probable, and worse ones less probable. The baseline does not change the gradient's expectation, but a good one (the average return from that state) can shrink its variance enormously.

For a linear-Gaussian policy a=θ⊤s+σεa = \theta^\top s + \sigma \varepsilon with ε∼N(0,1)\varepsilon \sim \mathcal{N}(0, 1), the score is

∇θlog⁡πθ(a∣s)=(a−θ⊤s) sσ2=ε sσ.\nabla_\theta \log \pi_\theta(a \mid s) = \frac{(a - \theta^\top s)\, s}{\sigma^2} = \frac{\varepsilon\, s}{\sigma}.

We test it on a problem whose answer we know. A 1 kg cart on a frictionless rail starts at rest somewhere in [−1,1][-1, 1] m and must get to x=0x = 0; the reward per 0.1 s step is −(x2+0.1u2) Δt-(x^2 + 0.1 u^2)\,\Delta t. This is an LQR problem, so the program also computes the optimal gain from the model with the Riccati iteration of State space and LQR. REINFORCE knows nothing of the model; it only sees states, its own random actions and rewards.

python
import numpy as np

# A 1 kg cart on a rail must reach x = 0 and stay there. State s = (x, v), action u = force (N).
DT, T, SIGMA = 0.1, 40, 0.5                      # 10 Hz control, 4 s episodes, exploration noise 0.5 N
A = np.array([[1.0, DT], [0.0, 1.0]])
B = np.array([0.0, DT])

def rollout(theta, n, rng, sigma=SIGMA):
    """n episodes in parallel with the linear-Gaussian policy u ~ N(theta . s, sigma^2)."""
    s = np.stack([rng.uniform(-1, 1, n), np.zeros(n)], axis=1)       # start anywhere in [-1, 1] m, at rest
    S, E, R = [], [], []
    for _ in range(T):
        eps = rng.standard_normal(n)
        u = s @ theta + sigma * eps
        R.append(-(s[:, 0] ** 2 + 0.1 * u ** 2) * DT)               # be near 0, without wasting effort
        S.append(s); E.append(eps)
        s = s @ A.T + np.outer(u, B)
    return np.array(S), np.array(E), np.array(R)                    # (T, n, 2), (T, n), (T, n)

def reinforce(iters=500, n=100, lr=0.3, seed=0):
    rng = np.random.default_rng(seed)
    theta = np.zeros(2)                                             # start knowing nothing: u = noise
    for it in range(iters):
        S, E, R = rollout(theta, n, rng)
        G = R[::-1].cumsum(axis=0)[::-1]                            # reward-to-go G_t
        adv = G - G.mean(axis=1, keepdims=True)                     # minus a baseline (batch mean at each t)
        # grad log pi(u|s) = (u - theta.s) s / sigma^2 = eps s / sigma
        grad = np.einsum("tn,tnk->k", adv * E / SIGMA, S) / n
        theta += lr * grad                                          # gradient ASCENT on expected return
        if it % 100 == 0 or it == iters - 1:
            print(f"update {it:3d}: mean return {R.sum(axis=0).mean():7.3f}   gains {np.round(theta, 2)}")
    return theta, iters * n

def lqr_gain(r=0.1):
    """The model-based answer for the same cost: iterate the discrete-time Riccati equation."""
    Qc, Rc, P = np.diag([1.0, 0.0]) * DT, r * DT, np.diag([1.0, 0.0]) * DT
    for _ in range(2000):
        K = (B @ P @ A) / (Rc + B @ P @ B)
        P = Qc + A.T @ P @ A - np.outer(A.T @ P @ B, K)
    return -K

theta, episodes = reinforce()
print(f"\nREINFORCE used {episodes} episodes = {episodes * T * DT / 3600:.0f} hours of driving at 10 Hz")
for name, gains in [("REINFORCE", theta), ("LQR from the model", lqr_gain())]:
    _, _, R = rollout(gains, 2000, np.random.default_rng(1), sigma=0.0)    # same 2000 starts, no noise
    print(f"{name:19s} gains {np.round(gains, 2)}: average return {R.sum(axis=0).mean():.4f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

REINFORCE finds gains of (−2.77,−2.54)(-2.77, -2.54), within a few per cent of the LQR gains (−2.79,−2.50)(-2.79, -2.50), and its average return without exploration noise, −0.2961-0.2961, matches LQR's −0.2960-0.2960 to three figures. It works, with no model at all. But look at what it cost: 50,000 episodes, the equivalent of 56 hours of continuous driving, to learn two numbers that the Riccati iteration computes from the model in a millisecond. The mean return printed during training also wanders up and down (from −0.408-0.408 to −0.431-0.431 and back) even as the gains improve, because each update sees only 100 noisy episodes. Sample inefficiency and high variance are the two facts of life of policy-gradient methods.

The comparison with LQR is fair only because the policy class contains the answer: the optimal controller for this problem is linear in (x,v)(x, v). Had we chosen u=θxu = \theta x alone, REINFORCE would have converged, just as confidently, to the best of a bad family. A learned policy can never beat the best member of the class it searches.

ExplanationH

Actor–critic, PPO and SAC

Modern deep RL algorithms keep the two ideas above — a value estimate learned by temporal differences, and a policy improved by gradient — and combine them.

Actor–critic. Replace the noisy Monte Carlo return in REINFORCE with a learned estimate. A critic Vϕ(s)V_\phi(s) is trained by TD learning; the actor πθ\pi_\theta is updated with the advantage, how much better an action was than the critic expected. The simplest one-step estimate is the TD error itself,

A^t=rt+1+γVϕ(st+1)−Vϕ(st),\hat{A}_t = r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t),

and generalised advantage estimation (GAE, Schulman and colleagues, 2015) blends one-step and many-step estimates with a parameter λ\lambda to trade bias against variance.

PPO. A policy-gradient step that is too large can wreck a policy in one update, and a robot's data distribution then changes under it. Proximal policy optimisation (Schulman and colleagues, 2017) reuses each batch of experience for several gradient steps but stops the policy moving far from the one that collected the data. With the probability ratio ρt(θ)=πθ(at∣st)/πθold(at∣st)\rho_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_\text{old}}(a_t \mid s_t), it maximises

LCLIP(θ)=Et[min⁡(ρt(θ) A^t, clip⁡(ρt(θ),1−ϵ,1+ϵ) A^t)],ϵ≈0.2.L^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t(\theta)\, \hat{A}_t,\ \operatorname{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon)\, \hat{A}_t\big)\Big], \qquad \epsilon \approx 0.2 .

Once the ratio leaves [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon] in the direction that would help, the objective goes flat and the gradient stops pushing. PPO is on-policy (it learns only from data from the current policy) and so needs a lot of experience, but it is simple, stable and parallelises perfectly. Paired with GPU simulators that step thousands of robots at once, it is the workhorse of locomotion: Nikita Rudin and colleagues at ETH Zurich trained a quadruped to walk on flat ground in under four minutes, and over rough terrain in about twenty, on a single workstation GPU (2021).

SAC. Soft actor–critic (Tuomas Haarnoja and colleagues, 2018) is off-policy: it stores every transition in a replay buffer and learns from all of them, which makes it far more economical with experience. It maximises reward plus the policy's entropy,

J(π)=∑tE[rt+1+α H(π(⋅∣st))],J(\pi) = \sum_t \mathbb{E}\big[r_{t+1} + \alpha\, \mathcal{H}(\pi(\cdot \mid s_t))\big],

which keeps the policy exploring and makes it robust; the temperature α\alpha is usually tuned automatically. Two critics are trained and the smaller of their estimates is used, to resist the over-optimism you saw in the Q-learning output. Haarnoja's team used it to teach a small quadruped to walk on the real floor in about two hours (2018).

On- or off-policyExperience neededTypical robot use
Tabular Q-learningoffmoderate, but only for tiny state spacesteaching, small discrete problems
REINFORCEonvery large; high varianceteaching, a baseline
PPOonlarge, cheap in parallel simulationlocomotion, sim-to-real policies
SAC (and relatives)offmuch smallerlearning on real hardware, manipulation

In practice nobody writes PPO from scratch for a project. The sketch below uses the open-source Stable-Baselines3 library and a Gymnasium environment; it needs PyTorch and is not run here. Robot-scale training uses GPU frameworks such as Isaac Lab, MuJoCo Playground or Genesis (Simulation), with the same algorithm inside.

python
# Not run here: needs PyTorch, stable-baselines3 and gymnasium installed.
import gymnasium as gym
from stable_baselines3 import PPO, SAC

env = gym.make("Pendulum-v1")                 # swing up and balance a pendulum
model = SAC("MlpPolicy", env, verbose=1)      # off-policy: comparatively few steps for this small task
model.learn(total_timesteps=20_000)
model.save("pendulum_sac")

model = PPO("MlpPolicy", env, verbose=1)      # on-policy: expect to need many more steps
model.learn(total_timesteps=500_000)
ComputationC

Writing rewards, and how they are gamed

The reward is the only description of the task the learner ever sees, and an optimiser will satisfy exactly what it says. A sparse reward (1 for success, 0 otherwise) says precisely what you want but is rarely reached by random exploration, so learning never starts. A dense reward (distance to the goal, forward velocity, a penalty on torque) gives a signal at every step but encodes your guesses about how to succeed, and the learner will exploit every gap between those guesses and your intent. This is reward hacking, also called specification gaming.

In a 2017 study of learning from human feedback, a simulated arm meant to grasp an object learned to put its hand between the camera and the object, so that to the people judging the video it only looked as if it were grasping (OpenAI). Simulated walkers rewarded for forward speed have discovered gaits that exploit numerical errors in the contact model, sliding or bouncing in ways no real robot could. Nothing is wrong with the optimiser in these cases; the specification was wrong.

Here is the commonest trap in robotics, reproduced in a corridor. A robot must drive to a charging dock nine cells away; docking pays 10. To speed up learning, an engineer adds a bonus of 1 "for every step that brings it closer to the dock" — and, as people often do, no penalty for steps that take it away. The program finds the optimal policy for each reward by value iteration, then drives it.

python
import numpy as np

# A corridor of 10 cells; the charging dock is cell 9. Actions: 0 = back, 1 = forward.
N, DOCK, GAMMA = 10, 9, 0.99

def step(s, a):
    return min(max(s + (1 if a == 1 else -1), 0), N - 1)

def dist(s):
    return DOCK - s

def task_reward(s, a, s2):
    return 10.0 if s2 == DOCK else 0.0               # what we actually want: reach the dock

def naive_bonus(s, a, s2):
    return 1.0 if dist(s2) < dist(s) else 0.0        # "reward progress", never penalise retreat

def potential_bonus(s, a, s2):
    phi = lambda x: -float(dist(x))                  # potential: minus the distance to the dock
    return GAMMA * phi(s2) - phi(s)                  # Ng, Harada and Russell's shaping term

def solve(reward):
    """Value iteration on the deterministic corridor; returns the greedy policy."""
    V = np.zeros(N)
    for _ in range(5000):
        Q = np.array([[reward(s, a, step(s, a)) + (0 if step(s, a) == DOCK else GAMMA * V[step(s, a)])
                       for a in (0, 1)] for s in range(N)])
        Q[DOCK] = 0.0
        V = Q.max(axis=1)
    return Q.argmax(axis=1), V

def drive(policy, steps=30):
    s, path = 0, [0]
    for _ in range(steps):
        s = step(s, policy[s])
        path.append(s)
        if s == DOCK:
            break
    return path

for name, extra in [("task reward only", None), ("+ naive progress bonus", naive_bonus),
                    ("+ potential-based bonus", potential_bonus)]:
    reward = task_reward if extra is None else (lambda s, a, s2, f=extra: task_reward(s, a, s2) + f(s, a, s2))
    policy, V = solve(reward)
    path = drive(policy)
    arrows = "".join("<>"[a] for a in policy[:DOCK]) + "D"
    print(f"{name:24s} policy {arrows}  V(start) = {V[0]:6.2f}  path: {path[:14]}{' ...' if len(path) > 14 else ''}"
          f"  -> {'docked in ' + str(len(path) - 1) + ' steps' if path[-1] == DOCK else 'never docks'}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With the naive bonus, the optimal policy drives to cell 8, one short of the dock, and then shuffles back and forth for ever: each forward step earns 1, each backward step costs nothing, and docking would end the stream of bonuses. Its value, 53.63, is far above the 18 or so that docking can offer. The robot is not broken; it is doing exactly what it was paid to do.

The third line uses potential-based shaping: choose any function Φ(s)\Phi(s) of the state — here minus the distance to the dock — and add F(s,s′)=γΦ(s′)−Φ(s)F(s, s') = \gamma \Phi(s') - \Phi(s) to the reward.

Theorem— Potential-based shaping (Ng, Harada and Russell, 1999)

Adding F(s,a,s′)=γΦ(s′)−Φ(s)F(s, a, s') = \gamma \Phi(s') - \Phi(s) to the reward of an MDP, for any bounded Φ\Phi with Φ=0\Phi = 0 at terminal states, changes every action value by the same state-dependent amount, Q′(s,a)=Q(s,a)−Φ(s)Q'(s, a) = Q(s, a) - \Phi(s). Hence it leaves the set of optimal policies unchanged.

Proof

Along any trajectory the added terms telescope: ∑kγk(γΦ(sk+1)−Φ(sk))=−Φ(s0)\sum_{k} \gamma^k \big(\gamma \Phi(s_{k+1}) - \Phi(s_k)\big) = -\Phi(s_0) when the trajectory ends in a terminal state with Φ=0\Phi = 0, or when γkΦ(sk)→0\gamma^k \Phi(s_k) \to 0. So every return starting from (s,a)(s, a) is shifted by exactly −Φ(s)-\Phi(s), whatever happens afterwards, and the action that maximises Q(s,⋅)Q(s, \cdot) also maximises Q′(s,⋅)Q'(s, \cdot).

∎

Check it in the output: V′(start)=18.23=9.23−Φ(0)=9.23+9V'(\text{start}) = 18.23 = 9.23 - \Phi(0) = 9.23 + 9. The shaped reward still guides exploration (each step towards the dock is now worth slightly more than one away), but it can no longer be farmed, because going round a loop returns Φ\Phi to where it started.

Practitioners combine several defences: start from the sparse task reward and add shaping terms one at a time; watch videos of the learned behaviour, not just the reward curve; penalise what you never want (joint limits, high impact forces, energy) explicitly; and terminate episodes on unsafe states. Recent work even writes reward functions with language models and refines them from training statistics (Eureka, NVIDIA and University of Pennsylvania, 2023), which moves the specification problem rather than removing it.

ExplanationH

Sample efficiency, safety, and why robots practise in simulation

Put the numbers side by side. Our two-parameter REINFORCE problem took 56 hours of experience. A locomotion policy with a neural network of a few hundred thousand parameters is typically trained with PPO on billions of simulated steps — at a 50 Hz control rate, 10910^9 steps is about 230 days of continuous walking. And every hour of real robot time costs more than an hour: batteries must be charged, the scene must be reset after each episode, motors heat up, and parts wear.

Exploration is also dangerous. A policy learns by trying things it has not tried before, and early in training those things are random. On a real robot that means hard contacts, joints driven to their limits, falls. The field's answers, in roughly increasing order of sophistication:

  • Hard limits outside the learner. Clip actions, limit joint velocities and torques in the motor drives, and run a safety controller that the policy cannot override; the policy's output is a request. Embedded control and safety covers how these layers are built.
  • Safety filters and constraints. Project each proposed action onto a set known to be safe (for example with a control barrier function), or solve a constrained MDP that maximises reward subject to a bound on expected cost.
  • Learning from data already collected. Offline RL trains on logged experience without exploring at all, and demonstrations can give the policy a sensible starting point (the next lesson).
  • Humans in the loop. Recent real-world systems let an operator take over when the robot is about to make a mistake, and learn from the corrections: HIL-SERL (UC Berkeley, 2024) reached near-perfect success on precise assembly tasks in 1 to 2.5 hours of training.
  • Simulation. Above all, learn somewhere that a fall costs nothing.

Simulation removes almost every obstacle at once. It runs faster than real time and in thousands of parallel copies on one GPU; it resets instantly; it never breaks; and it can give the learner information a real robot does not have, such as exact contact forces or terrain heights, which the lesson after next exploits. The cost is the reality gap: a policy is only as good as the simulator it practised in, and reinforcement learning is superb at finding and exploiting a simulator's mistakes.

The result is a recognisable division of labour, as of 2026. Locomotion for quadrupeds and humanoids is trained almost entirely in simulation with PPO and transferred to hardware; examples include ETH Zurich's ANYmal, quadrupeds and humanoids from Unitree in China, and research bipeds at Oregon State and UC Berkeley. Agile flight followed the same route: the University of Zurich's Swift drone, trained in simulation, won most of its head-to-head races against three human champions in 2023. Real-world RL survives where simulation is hardest — contact-rich manipulation with deformable or unusual objects — and there it is usually off-policy, started from demonstrations, and supervised: Google's QT-Opt (2018) learned vision-based grasping from over 580,000 real grasp attempts collected by a fleet of arms. Manipulation more broadly has leaned towards learning from demonstrations, the subject of the next lesson, often with RL used afterwards to fine-tune. Tang and colleagues' 2024 survey gives a careful map of where deep RL has and has not worked on real robots.

If you want to try RL on hardware yourself, do it the way professionals do: in simulation first, then on a small, slow, battery-powered robot, with low torque limits, a clear space around it, and a person ready at the stop button.

Common misconceptionsH

Common misconceptions

Misconception

"Reinforcement learning means the robot learns by itself, so you don't need to understand the task." Someone chose the state, the action space, the reward, the termination conditions, the policy class and the simulator. Each is a design decision with as much influence on the result as the algorithm, and most RL failures are failures of one of them.

Misconception

"A rising reward curve means the robot is getting better at the task." It means it is getting better at the reward. The corridor robot's return rose to 53.63 while it never docked. Always watch the behaviour and track a separate, honest success measure.

Misconception

"RL is an alternative to control theory." It is a way of solving optimal-control problems from samples instead of from equations; the Bellman equation and the Riccati equation are the same idea. Deployed learned policies usually sit on top of classical loops: a locomotion policy outputs joint targets at 50 Hz, and PD loops in the drives track them at kilohertz rates.

Misconception

"The value estimate tells you how good the policy is." Only if the estimate is accurate, which it is not early in training or in rarely visited states. Our Q-learner believed −3.41-3.41 for a policy worth −19.91-19.91. Evaluate policies by running them.

Exercises

Exercises

1. Reinforcement learning for robot control

Warm-upHuman

A robot receives rewards r1=4r_1 = 4, r2=8r_2 = 8, r3=16r_3 = 16 and nothing afterwards. With discount γ=0.5\gamma = 0.5, what is the return G0=r1+γr2+γ2r3G_0 = r_1 + \gamma r_2 + \gamma^2 r_3?

2. Reinforcement learning for robot control

Warm-upHuman

A Q-learner has Q(s,a)=2Q(s, a) = 2. It takes aa, receives r=−1r = -1 and lands in s′s', where max⁡a′Q(s′,a′)=10\max_{a'} Q(s', a') = 10. With γ=0.9\gamma = 0.9 and learning rate α=0.5\alpha = 0.5, what is the new value of Q(s,a)Q(s, a)?

3. Reinforcement learning for robot control

CoreComputational

Write rewards_to_go(rewards, gamma) that takes a list of rewards r1,…,rTr_1, \ldots, r_T from one episode and returns a NumPy array whose entry tt (counting from 0) is Gt=∑k≥0γkrt+k+1G_t = \sum_{k \ge 0} \gamma^k r_{t+k+1}, the discounted reward collected from step tt onwards. Do it in one backward pass, using Gt=rt+1+γGt+1G_t = r_{t+1} + \gamma G_{t+1}.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

4. Reinforcement learning for robot control

CoreComputational

Write q_update(Q, s, a, r, s2, done, alpha, gamma). Q is a NumPy array of shape (n_states, n_actions). Apply one Q-learning update to Q[s, a] in place for the transition (s,a,r,s′)(s, a, r, s') and return the TD error. If done is true, s′s' is terminal and its value counts as 0.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

5. Reinforcement learning for robot control

CoreComputational

For the linear-Gaussian policy πθ(a∣s)=N(a; θ⊤s, σ2)\pi_\theta(a \mid s) = \mathcal{N}(a;\ \theta^\top s,\ \sigma^2), write log_pi(theta, s, a, sigma) returning log⁡πθ(a∣s)\log \pi_\theta(a \mid s) and score(theta, s, a, sigma) returning the gradient ∇θlog⁡πθ(a∣s)\nabla_\theta \log \pi_\theta(a \mid s) as a NumPy array. The tests compare your score with a finite-difference derivative of your log_pi.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

6. Reinforcement learning for robot control

CoreHuman

Which of these algorithms can learn correctly from a replay buffer of transitions collected long ago by older versions of the policy? Select all that apply.

Choose all that apply

7. Reinforcement learning for robot control

CoreHuman

A pushing robot is rewarded +10+10 when the box reaches the goal. Which extra reward term is guaranteed not to change which policies are optimal?

Choose one

8. Reinforcement learning for robot control

StretchHuman

A PPO run in simulation used 3×1083 \times 10^8 environment steps at a control rate of 50 Hz. If the same experience had to be collected on one real robot running non-stop (no resets, no charging), how many whole days would it take? Round down.

Further reading and resources