A controller nobody wrote
Everything you have built so far in this course started from a model. You wrote down the dynamics of a joint and tuned a PID loop for it; you linearised a cart-pole and solved a Riccati equation for its LQR gain; you optimised trajectories through a model. When the model is good, these methods are hard to beat: they are fast, predictable and come with guarantees.
Some tasks resist this. A quadruped crossing loose gravel touches the ground with four feet at unpredictable moments; a hand turning a cube between its fingers makes and breaks a dozen contacts a second. Writing an accurate model of such a task is hard, and designing a controller from it is harder still. Yet it is easy to score an attempt: did the robot move forward without falling? Is the cube in the target orientation?
Reinforcement learning (RL) turns that score into a controller. The robot tries something, receives a number called a reward, and adjusts its behaviour to collect more reward in future. Nobody writes the control law; it is found by trial and error. Since about 2019 this has produced the walking controllers on many research and commercial quadrupeds (see Legged locomotion), a drone that beat human champions at racing (University of Zurich, 2023), and in-hand manipulation that hand-designed controllers had never achieved.
It also has a price: RL needs enormous amounts of experience, exploring can break things, and a sensible-looking reward can be satisfied in ways you never intended. You will implement the two families every modern algorithm builds on — Q-learning and policy gradients — and see why robot RL is done in a simulator first. For gradient descent, networks and training practice, see Lumenary CS: Learning from data, Neural networks and Training in practice.
Markov decision processes
RL formalises a robot task as a loop between an agent (the controller being learned) and an environment (the robot's body and everything around it).
Definition— Markov decision process (MDP)
An MDP is a tuple : a set of states , a set of actions , transition probabilities , a reward function , and a discount factor . It is Markov because the next state depends only on the current state and action, not on how the robot got there.
A policy gives the probability of each action in each state (a deterministic policy is the special case ). Starting from and following produces a trajectory . Its return from time is the discounted sum
The state-value function and the action-value function of are the expected returns
The goal is a policy whose value is as large as possible in every state; its value functions are written and .
The discount keeps infinite sums finite and sets a horizon: with a reward 100 steps away counts for of an immediate one, so the robot plans roughly steps ahead. At a 50 Hz control rate that is two seconds, about right for a stride or a grasp.
Value functions satisfy a consistency condition, which every algorithm in this lesson exploits. The return is the first reward plus the discounted return from the next state, , so taking expectations gives the Bellman equations
The second, the Bellman optimality equation, says: the value of doing now is what you get immediately plus the value of acting optimally afterwards. Given , the optimal policy is simply to pick in every state.
Q-learning on a warehouse floor
If we knew and we could find by value iteration: start from any , and repeatedly replace it with the right-hand side of the Bellman optimality equation. The update is a contraction (each sweep shrinks the largest error by a factor ), so it converges.
Q-learning (Christopher Watkins, 1989) does the same without the model. Each time the robot takes action in state and observes reward and next state , it nudges its estimate towards a one-sample version of the Bellman target:
The bracket is the temporal-difference (TD) error: how much better or worse things turned out than predicted. With a small learning rate the noise of individual samples averages out. To explore, the robot usually acts greedily but picks a random action with probability (-greedy). Q-learning is off-policy: it learns the value of the greedy policy even while behaving partly at random, because the target uses rather than the action actually taken next.
Our robot is a small delivery cart in a warehouse aisle. It starts at S and must reach the charging dock G. Shelving (#) blocks the row above the aisle; drain grates (X) line the row below, and a wheel that drops into a grate ends the episode (someone has to come and lift the cart out). Each move costs , reaching the dock pays , a grate costs . The floor is slippery: with probability 0.2 the cart slides sideways instead of going where it was told.
The program solves the problem twice: once by value iteration, which is allowed to read the model, and once by Q-learning, which only gets to drive the cart and see what happens. To judge Q-learning fairly, it also computes the true expected return of each learned policy exactly, by solving the linear Bellman equations for that policy.
import numpy as np
# A warehouse aisle seen from above: S start, G charging dock, # shelving, X drain grates.
MAP = ["......",
".####.",
"S....G",
".XXXX.",
"......"]
MOVES, ARROWS = [(-1, 0), (0, 1), (1, 0), (0, -1)], "^>v<" # north, east, south, west
GAMMA = 0.95
cells = [(r, c) for r in range(5) for c in range(6) if MAP[r][c] != "#"]
index = {cell: i for i, cell in enumerate(cells)}
START = index[(2, 0)]
TERMINAL = {i for i, (r, c) in enumerate(cells) if MAP[r][c] in "GX"}
def outcomes(s, a, slip):
"""(probability, next state, reward) for action a in state s. The wheels slip sideways with probability slip."""
out = []
for p, a2 in [(1 - slip, a), (slip / 2, (a + 1) % 4), (slip / 2, (a + 3) % 4)]:
r, c = cells[s][0] + MOVES[a2][0], cells[s][1] + MOVES[a2][1]
s2 = index.get((r, c), s) # a wall or shelf: stay where you are
reward = -1.0 + {"G": 10.0, "X": -20.0}.get(MAP[cells[s2][0]][cells[s2][1]], 0.0)
out.append((p, s2, reward))
return out
def value_iteration(slip):
"""The model-based answer: iterate the Bellman optimality equation until Q stops changing."""
Q = np.zeros((len(cells), 4))
for _ in range(500):
V = np.where([s in TERMINAL for s in range(len(cells))], 0.0, Q.max(axis=1))
Q = np.array([[sum(p * (r + GAMMA * V[s2]) for p, s2, r in outcomes(s, a, slip)) for a in range(4)]
for s in range(len(cells))])
return Q
def q_learning(episodes, slip, alpha=0.1, eps=0.2, seed=0):
"""The model-free answer: learn Q from experience alone, one transition at a time."""
rng = np.random.default_rng(seed)
Q = np.zeros((len(cells), 4))
for _ in range(episodes):
s = START
for _ in range(200):
a = int(rng.integers(4)) if rng.random() < eps else int(np.argmax(Q[s])) # epsilon-greedy
outs = outcomes(s, a, slip)
_, s2, r = outs[rng.choice(3, p=[o[0] for o in outs])] # the world decides
target = r if s2 in TERMINAL else r + GAMMA * Q[s2].max() # TD target
Q[s, a] += alpha * (target - Q[s, a])
s = s2
if s in TERMINAL:
break
return Q
def true_value(policy, slip):
"""Exact expected return of a fixed policy: solve the linear Bellman equations V = R + gamma P V."""
n = len(cells)
P, R = np.zeros((n, n)), np.zeros(n)
for s in set(range(n)) - TERMINAL:
for p, s2, r in outcomes(s, policy[s], slip):
P[s, s2] += p
R[s] += p * r
return np.linalg.solve(np.eye(n) - GAMMA * P, R)
def show(Q, title):
print(title)
for r in range(5):
print(" " + " ".join(MAP[r][c] if MAP[r][c] in "#GX" else ARROWS[int(np.argmax(Q[index[(r, c)]]))]
for c in range(6)))
show(value_iteration(slip=0.0), "optimal policy, no slip:")
Qstar = value_iteration(slip=0.2)
show(Qstar, "optimal policy, 20% sideways slip:")
print(f"optimal value of the start cell: {Qstar[START].max():.2f}\n")
for episodes in [50, 200, 1000, 5000]:
Q = q_learning(episodes, slip=0.2)
a = int(np.argmax(Q[START]))
V = true_value(Q.argmax(axis=1), slip=0.2)
print(f"Q-learning, {episodes:4d} episodes: first move {ARROWS[a]}, its estimate {Q[START, a]:6.2f}, "
f"true value of its policy {V[START]:6.2f}")
show(Q, "\npolicy learned by Q-learning (5000 episodes):")Read the output carefully; it contains most of the lessons of RL in miniature.
- The optimal policy depends on the noise. Without slip, the best first move from S is east, straight along the aisle. With 20% slip the best move is north: four eastward steps beside the grates would each risk a 10% slide into one, and the detour of four extra steps is cheaper. No one told the robot about risk; the expected-return objective produced it.
- Estimates can be confidently wrong. After 50 episodes Q-learning believes its first move is worth , close to the true optimum of , but the policy it would actually follow is worth : it heads south and down a route that ends in a grate far too often. A value estimate is only as good as the experience behind it.
- It gets there. From 200 episodes on, the learned policy starts by going north, and after 5,000 episodes its true value is the optimal .
- Only visited states are learned. In the bottom row, which a good policy never visits, Q-learning's arrows are still wrong after 5,000 episodes. That is harmless here and alarming on a real robot: a learned policy may do something arbitrary in a state it rarely saw during training.
Policy gradients
Robot actions are continuous: torques, velocities, joint targets. Taking over a continuous action space is itself an optimisation problem, so for control it is often simpler to parametrise the policy directly, as , and climb the gradient of the expected return
where is a whole trajectory. The difficulty is that changes which trajectories happen, through dynamics we cannot differentiate. A short identity gets around it.
Theorem— Policy gradient (REINFORCE)
For a policy that is differentiable in , and any baseline that does not depend on the action,
where is the (discounted) reward collected from time onwards.
Proof
Write for the probability of a trajectory and for its total reward. Since ,
Now , and the dynamics do not depend on , so : the unknown model drops out. Rewards received before time cannot be influenced by , and their products with the score average to zero, which leaves only the reward-to-go . Finally, , so subtracting any action-independent leaves the expectation unchanged.
∎
The estimator says: make actions that were followed by better-than-baseline returns more probable, and worse ones less probable. The baseline does not change the gradient's expectation, but a good one (the average return from that state) can shrink its variance enormously.
For a linear-Gaussian policy with , the score is
We test it on a problem whose answer we know. A 1 kg cart on a frictionless rail starts at rest somewhere in m and must get to ; the reward per 0.1 s step is . This is an LQR problem, so the program also computes the optimal gain from the model with the Riccati iteration of State space and LQR. REINFORCE knows nothing of the model; it only sees states, its own random actions and rewards.
import numpy as np
# A 1 kg cart on a rail must reach x = 0 and stay there. State s = (x, v), action u = force (N).
DT, T, SIGMA = 0.1, 40, 0.5 # 10 Hz control, 4 s episodes, exploration noise 0.5 N
A = np.array([[1.0, DT], [0.0, 1.0]])
B = np.array([0.0, DT])
def rollout(theta, n, rng, sigma=SIGMA):
"""n episodes in parallel with the linear-Gaussian policy u ~ N(theta . s, sigma^2)."""
s = np.stack([rng.uniform(-1, 1, n), np.zeros(n)], axis=1) # start anywhere in [-1, 1] m, at rest
S, E, R = [], [], []
for _ in range(T):
eps = rng.standard_normal(n)
u = s @ theta + sigma * eps
R.append(-(s[:, 0] ** 2 + 0.1 * u ** 2) * DT) # be near 0, without wasting effort
S.append(s); E.append(eps)
s = s @ A.T + np.outer(u, B)
return np.array(S), np.array(E), np.array(R) # (T, n, 2), (T, n), (T, n)
def reinforce(iters=500, n=100, lr=0.3, seed=0):
rng = np.random.default_rng(seed)
theta = np.zeros(2) # start knowing nothing: u = noise
for it in range(iters):
S, E, R = rollout(theta, n, rng)
G = R[::-1].cumsum(axis=0)[::-1] # reward-to-go G_t
adv = G - G.mean(axis=1, keepdims=True) # minus a baseline (batch mean at each t)
# grad log pi(u|s) = (u - theta.s) s / sigma^2 = eps s / sigma
grad = np.einsum("tn,tnk->k", adv * E / SIGMA, S) / n
theta += lr * grad # gradient ASCENT on expected return
if it % 100 == 0 or it == iters - 1:
print(f"update {it:3d}: mean return {R.sum(axis=0).mean():7.3f} gains {np.round(theta, 2)}")
return theta, iters * n
def lqr_gain(r=0.1):
"""The model-based answer for the same cost: iterate the discrete-time Riccati equation."""
Qc, Rc, P = np.diag([1.0, 0.0]) * DT, r * DT, np.diag([1.0, 0.0]) * DT
for _ in range(2000):
K = (B @ P @ A) / (Rc + B @ P @ B)
P = Qc + A.T @ P @ A - np.outer(A.T @ P @ B, K)
return -K
theta, episodes = reinforce()
print(f"\nREINFORCE used {episodes} episodes = {episodes * T * DT / 3600:.0f} hours of driving at 10 Hz")
for name, gains in [("REINFORCE", theta), ("LQR from the model", lqr_gain())]:
_, _, R = rollout(gains, 2000, np.random.default_rng(1), sigma=0.0) # same 2000 starts, no noise
print(f"{name:19s} gains {np.round(gains, 2)}: average return {R.sum(axis=0).mean():.4f}")REINFORCE finds gains of , within a few per cent of the LQR gains , and its average return without exploration noise, , matches LQR's to three figures. It works, with no model at all. But look at what it cost: 50,000 episodes, the equivalent of 56 hours of continuous driving, to learn two numbers that the Riccati iteration computes from the model in a millisecond. The mean return printed during training also wanders up and down (from to and back) even as the gains improve, because each update sees only 100 noisy episodes. Sample inefficiency and high variance are the two facts of life of policy-gradient methods.
The comparison with LQR is fair only because the policy class contains the answer: the optimal controller for this problem is linear in . Had we chosen alone, REINFORCE would have converged, just as confidently, to the best of a bad family. A learned policy can never beat the best member of the class it searches.
Actor–critic, PPO and SAC
Modern deep RL algorithms keep the two ideas above — a value estimate learned by temporal differences, and a policy improved by gradient — and combine them.
Actor–critic. Replace the noisy Monte Carlo return in REINFORCE with a learned estimate. A critic is trained by TD learning; the actor is updated with the advantage, how much better an action was than the critic expected. The simplest one-step estimate is the TD error itself,
and generalised advantage estimation (GAE, Schulman and colleagues, 2015) blends one-step and many-step estimates with a parameter to trade bias against variance.
PPO. A policy-gradient step that is too large can wreck a policy in one update, and a robot's data distribution then changes under it. Proximal policy optimisation (Schulman and colleagues, 2017) reuses each batch of experience for several gradient steps but stops the policy moving far from the one that collected the data. With the probability ratio , it maximises
Once the ratio leaves in the direction that would help, the objective goes flat and the gradient stops pushing. PPO is on-policy (it learns only from data from the current policy) and so needs a lot of experience, but it is simple, stable and parallelises perfectly. Paired with GPU simulators that step thousands of robots at once, it is the workhorse of locomotion: Nikita Rudin and colleagues at ETH Zurich trained a quadruped to walk on flat ground in under four minutes, and over rough terrain in about twenty, on a single workstation GPU (2021).
SAC. Soft actor–critic (Tuomas Haarnoja and colleagues, 2018) is off-policy: it stores every transition in a replay buffer and learns from all of them, which makes it far more economical with experience. It maximises reward plus the policy's entropy,
which keeps the policy exploring and makes it robust; the temperature is usually tuned automatically. Two critics are trained and the smaller of their estimates is used, to resist the over-optimism you saw in the Q-learning output. Haarnoja's team used it to teach a small quadruped to walk on the real floor in about two hours (2018).
| On- or off-policy | Experience needed | Typical robot use | |
|---|---|---|---|
| Tabular Q-learning | off | moderate, but only for tiny state spaces | teaching, small discrete problems |
| REINFORCE | on | very large; high variance | teaching, a baseline |
| PPO | on | large, cheap in parallel simulation | locomotion, sim-to-real policies |
| SAC (and relatives) | off | much smaller | learning on real hardware, manipulation |
In practice nobody writes PPO from scratch for a project. The sketch below uses the open-source Stable-Baselines3 library and a Gymnasium environment; it needs PyTorch and is not run here. Robot-scale training uses GPU frameworks such as Isaac Lab, MuJoCo Playground or Genesis (Simulation), with the same algorithm inside.
# Not run here: needs PyTorch, stable-baselines3 and gymnasium installed.
import gymnasium as gym
from stable_baselines3 import PPO, SAC
env = gym.make("Pendulum-v1") # swing up and balance a pendulum
model = SAC("MlpPolicy", env, verbose=1) # off-policy: comparatively few steps for this small task
model.learn(total_timesteps=20_000)
model.save("pendulum_sac")
model = PPO("MlpPolicy", env, verbose=1) # on-policy: expect to need many more steps
model.learn(total_timesteps=500_000)Writing rewards, and how they are gamed
The reward is the only description of the task the learner ever sees, and an optimiser will satisfy exactly what it says. A sparse reward (1 for success, 0 otherwise) says precisely what you want but is rarely reached by random exploration, so learning never starts. A dense reward (distance to the goal, forward velocity, a penalty on torque) gives a signal at every step but encodes your guesses about how to succeed, and the learner will exploit every gap between those guesses and your intent. This is reward hacking, also called specification gaming.
In a 2017 study of learning from human feedback, a simulated arm meant to grasp an object learned to put its hand between the camera and the object, so that to the people judging the video it only looked as if it were grasping (OpenAI). Simulated walkers rewarded for forward speed have discovered gaits that exploit numerical errors in the contact model, sliding or bouncing in ways no real robot could. Nothing is wrong with the optimiser in these cases; the specification was wrong.
Here is the commonest trap in robotics, reproduced in a corridor. A robot must drive to a charging dock nine cells away; docking pays 10. To speed up learning, an engineer adds a bonus of 1 "for every step that brings it closer to the dock" — and, as people often do, no penalty for steps that take it away. The program finds the optimal policy for each reward by value iteration, then drives it.
import numpy as np
# A corridor of 10 cells; the charging dock is cell 9. Actions: 0 = back, 1 = forward.
N, DOCK, GAMMA = 10, 9, 0.99
def step(s, a):
return min(max(s + (1 if a == 1 else -1), 0), N - 1)
def dist(s):
return DOCK - s
def task_reward(s, a, s2):
return 10.0 if s2 == DOCK else 0.0 # what we actually want: reach the dock
def naive_bonus(s, a, s2):
return 1.0 if dist(s2) < dist(s) else 0.0 # "reward progress", never penalise retreat
def potential_bonus(s, a, s2):
phi = lambda x: -float(dist(x)) # potential: minus the distance to the dock
return GAMMA * phi(s2) - phi(s) # Ng, Harada and Russell's shaping term
def solve(reward):
"""Value iteration on the deterministic corridor; returns the greedy policy."""
V = np.zeros(N)
for _ in range(5000):
Q = np.array([[reward(s, a, step(s, a)) + (0 if step(s, a) == DOCK else GAMMA * V[step(s, a)])
for a in (0, 1)] for s in range(N)])
Q[DOCK] = 0.0
V = Q.max(axis=1)
return Q.argmax(axis=1), V
def drive(policy, steps=30):
s, path = 0, [0]
for _ in range(steps):
s = step(s, policy[s])
path.append(s)
if s == DOCK:
break
return path
for name, extra in [("task reward only", None), ("+ naive progress bonus", naive_bonus),
("+ potential-based bonus", potential_bonus)]:
reward = task_reward if extra is None else (lambda s, a, s2, f=extra: task_reward(s, a, s2) + f(s, a, s2))
policy, V = solve(reward)
path = drive(policy)
arrows = "".join("<>"[a] for a in policy[:DOCK]) + "D"
print(f"{name:24s} policy {arrows} V(start) = {V[0]:6.2f} path: {path[:14]}{' ...' if len(path) > 14 else ''}"
f" -> {'docked in ' + str(len(path) - 1) + ' steps' if path[-1] == DOCK else 'never docks'}")With the naive bonus, the optimal policy drives to cell 8, one short of the dock, and then shuffles back and forth for ever: each forward step earns 1, each backward step costs nothing, and docking would end the stream of bonuses. Its value, 53.63, is far above the 18 or so that docking can offer. The robot is not broken; it is doing exactly what it was paid to do.
The third line uses potential-based shaping: choose any function of the state — here minus the distance to the dock — and add to the reward.
Theorem— Potential-based shaping (Ng, Harada and Russell, 1999)
Adding to the reward of an MDP, for any bounded with at terminal states, changes every action value by the same state-dependent amount, . Hence it leaves the set of optimal policies unchanged.
Proof
Along any trajectory the added terms telescope: when the trajectory ends in a terminal state with , or when . So every return starting from is shifted by exactly , whatever happens afterwards, and the action that maximises also maximises .
∎
Check it in the output: . The shaped reward still guides exploration (each step towards the dock is now worth slightly more than one away), but it can no longer be farmed, because going round a loop returns to where it started.
Practitioners combine several defences: start from the sparse task reward and add shaping terms one at a time; watch videos of the learned behaviour, not just the reward curve; penalise what you never want (joint limits, high impact forces, energy) explicitly; and terminate episodes on unsafe states. Recent work even writes reward functions with language models and refines them from training statistics (Eureka, NVIDIA and University of Pennsylvania, 2023), which moves the specification problem rather than removing it.
Sample efficiency, safety, and why robots practise in simulation
Put the numbers side by side. Our two-parameter REINFORCE problem took 56 hours of experience. A locomotion policy with a neural network of a few hundred thousand parameters is typically trained with PPO on billions of simulated steps — at a 50 Hz control rate, steps is about 230 days of continuous walking. And every hour of real robot time costs more than an hour: batteries must be charged, the scene must be reset after each episode, motors heat up, and parts wear.
Exploration is also dangerous. A policy learns by trying things it has not tried before, and early in training those things are random. On a real robot that means hard contacts, joints driven to their limits, falls. The field's answers, in roughly increasing order of sophistication:
- Hard limits outside the learner. Clip actions, limit joint velocities and torques in the motor drives, and run a safety controller that the policy cannot override; the policy's output is a request. Embedded control and safety covers how these layers are built.
- Safety filters and constraints. Project each proposed action onto a set known to be safe (for example with a control barrier function), or solve a constrained MDP that maximises reward subject to a bound on expected cost.
- Learning from data already collected. Offline RL trains on logged experience without exploring at all, and demonstrations can give the policy a sensible starting point (the next lesson).
- Humans in the loop. Recent real-world systems let an operator take over when the robot is about to make a mistake, and learn from the corrections: HIL-SERL (UC Berkeley, 2024) reached near-perfect success on precise assembly tasks in 1 to 2.5 hours of training.
- Simulation. Above all, learn somewhere that a fall costs nothing.
Simulation removes almost every obstacle at once. It runs faster than real time and in thousands of parallel copies on one GPU; it resets instantly; it never breaks; and it can give the learner information a real robot does not have, such as exact contact forces or terrain heights, which the lesson after next exploits. The cost is the reality gap: a policy is only as good as the simulator it practised in, and reinforcement learning is superb at finding and exploiting a simulator's mistakes.
The result is a recognisable division of labour, as of 2026. Locomotion for quadrupeds and humanoids is trained almost entirely in simulation with PPO and transferred to hardware; examples include ETH Zurich's ANYmal, quadrupeds and humanoids from Unitree in China, and research bipeds at Oregon State and UC Berkeley. Agile flight followed the same route: the University of Zurich's Swift drone, trained in simulation, won most of its head-to-head races against three human champions in 2023. Real-world RL survives where simulation is hardest — contact-rich manipulation with deformable or unusual objects — and there it is usually off-policy, started from demonstrations, and supervised: Google's QT-Opt (2018) learned vision-based grasping from over 580,000 real grasp attempts collected by a fleet of arms. Manipulation more broadly has leaned towards learning from demonstrations, the subject of the next lesson, often with RL used afterwards to fine-tune. Tang and colleagues' 2024 survey gives a careful map of where deep RL has and has not worked on real robots.
If you want to try RL on hardware yourself, do it the way professionals do: in simulation first, then on a small, slow, battery-powered robot, with low torque limits, a clear space around it, and a person ready at the stop button.
Common misconceptions
Misconception
"Reinforcement learning means the robot learns by itself, so you don't need to understand the task." Someone chose the state, the action space, the reward, the termination conditions, the policy class and the simulator. Each is a design decision with as much influence on the result as the algorithm, and most RL failures are failures of one of them.
Misconception
"A rising reward curve means the robot is getting better at the task." It means it is getting better at the reward. The corridor robot's return rose to 53.63 while it never docked. Always watch the behaviour and track a separate, honest success measure.
Misconception
"RL is an alternative to control theory." It is a way of solving optimal-control problems from samples instead of from equations; the Bellman equation and the Riccati equation are the same idea. Deployed learned policies usually sit on top of classical loops: a locomotion policy outputs joint targets at 50 Hz, and PD loops in the drives track them at kilohertz rates.
Misconception
"The value estimate tells you how good the policy is." Only if the estimate is accurate, which it is not early in training or in rarely visited states. Our Q-learner believed for a policy worth . Evaluate policies by running them.
Exercises
1. Reinforcement learning for robot control
Warm-upHumanA robot receives rewards , , and nothing afterwards. With discount , what is the return ?
2. Reinforcement learning for robot control
Warm-upHumanA Q-learner has . It takes , receives and lands in , where . With and learning rate , what is the new value of ?
3. Reinforcement learning for robot control
CoreComputationalWrite rewards_to_go(rewards, gamma) that takes a list of rewards from one episode and returns a NumPy array whose entry (counting from 0) is , the discounted reward collected from step onwards. Do it in one backward pass, using .
4. Reinforcement learning for robot control
CoreComputationalWrite q_update(Q, s, a, r, s2, done, alpha, gamma). Q is a NumPy array of shape (n_states, n_actions). Apply one Q-learning update to Q[s, a] in place for the transition and return the TD error. If done is true, is terminal and its value counts as 0.
5. Reinforcement learning for robot control
CoreComputationalFor the linear-Gaussian policy , write log_pi(theta, s, a, sigma) returning and score(theta, s, a, sigma) returning the gradient as a NumPy array. The tests compare your score with a finite-difference derivative of your log_pi.
6. Reinforcement learning for robot control
CoreHumanWhich of these algorithms can learn correctly from a replay buffer of transitions collected long ago by older versions of the policy? Select all that apply.
7. Reinforcement learning for robot control
CoreHumanA pushing robot is rewarded when the box reaches the goal. Which extra reward term is guaranteed not to change which policies are optimal?
8. Reinforcement learning for robot control
StretchHumanA PPO run in simulation used environment steps at a control rate of 50 Hz. If the same experience had to be collected on one real robot running non-stop (no resets, no charging), how many whole days would it take? Round down.
Further reading and resources
- Reinforcement Learning: An Introduction (2nd edition) · Richard S. Sutton and Andrew G. Barto · 2018The standard textbook (MIT Press); the authors make the full text freely available on Richard Sutton's website.
- Q-learning · Christopher J. C. H. Watkins and Peter Dayan · 1992The convergence proof for tabular Q-learning, from Watkins's 1989 thesis.
- Simple statistical gradient-following algorithms for connectionist reinforcement learning · Ronald J. Williams · 1992The REINFORCE family of policy-gradient algorithms.
- Human-level control through deep reinforcement learning · Volodymyr Mnih et al. (DeepMind) · 2015DQN, Q-learning with a neural network, replay buffer and target network.
- High-Dimensional Continuous Control Using Generalized Advantage Estimation · John Schulman et al. · 2015
- Proximal Policy Optimization Algorithms · John Schulman et al. · 2017
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor · Tuomas Haarnoja et al. · 2018
- Learning to Walk via Deep Reinforcement Learning · Tuomas Haarnoja et al. · 2018A small quadruped learning a gait from scratch on the real floor, in about two hours.
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation · Dmitry Kalashnikov et al. · 2018Off-policy RL from over 580,000 real grasp attempts.
- Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning · Nikita Rudin et al. · 2021
- Champion-level drone racing using deep reinforcement learning · Elia Kaufmann et al. · 2023The Swift system from the University of Zurich, trained in simulation, racing human champions.
- Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning · Jianlan Luo et al. · 2024Real-world RL with human corrections, reaching high success on precise tasks in 1 to 2.5 hours of training.
- Policy invariance under reward transformations: theory and application to reward shaping · Andrew Y. Ng, Daishi Harada and Stuart Russell · 1999Proceedings of ICML 1999. Proves that potential-based shaping never changes the optimal policy.
- Concrete Problems in AI Safety · Dario Amodei et al. · 2016Reward hacking, safe exploration and other failure modes, with robot examples.
- Learning from human preferences · OpenAI · 2017Includes the simulated arm that learned to look as if it were grasping.
- Eureka: Human-Level Reward Design via Coding Large Language Models · Yecheng Jason Ma et al. · 2023
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes · Chen Tang et al. · 2024
- Spinning Up in Deep RL · OpenAIA free, clear introduction to the deep RL algorithms in this lesson, with reference code.
- Stable-Baselines3Reliable open-source implementations of PPO, SAC and others.
- Gymnasium · Farama FoundationThe standard interface between RL algorithms and environments.