Skip to content
LumenaryAcademy
Frontier 85 min

Vision-language-action models

How a vision-language model learns to move a robot — action tokens and their error, flow-matching action experts, co-training on web and robot data — and what the generalist policies of 2026 can and cannot do.

Code tested · 5AI-draftedAwaiting review by a roboticist

Your guides: Chloe · The Builder, Margaret · The Experimenter, Gideon · The Safety Engineer

MotivationH

From describing a scene to acting in it

Show a modern vision-language model (VLM) a photograph of a kitchen counter and ask what is there, and it will tell you: a chipped blue mug, a sponge, a bottle of washing-up liquid, and that the mug probably belongs in the sink. Ask it which way to move a robot's wrist, in metres, for the next tenth of a second, and it has no idea. Its knowledge came from billions of images and pages of text, and none of them contained a joint angle.

The policies of Imitation learning have the opposite problem. Trained on a few hundred demonstrations of one task, a diffusion policy can fold a towel with real finesse, but swap the towel for a T-shirt, or ask it in words to do something else, and it is lost. It knows how to move and almost nothing about the world.

A vision-language-action model (VLA) tries to have both. Start from a VLM pretrained on the web, keep training it on robot demonstrations until it outputs motor commands, and hope the semantic knowledge carries over: that a robot that has never been taught the word "dinosaur" can still pick up the toy dinosaur, because the VLM knew what one looks like. The bet was made publicly by Google's RT-2 in July 2023, and it paid off well enough that by 2026 every major robot-learning lab, and most well-funded robot companies, trains one.

This lesson takes the VLA apart. We will look at its anatomy, implement the two main ways of producing actions — discrete tokens and continuous flow-matching heads — measure what each costs, then turn to how these models are trained and what, honestly, they can and cannot do as of late 2026. It assumes the transformer of Attention and the transformer and the pretraining story of Language models, and the datasets of Robot data at scale.

DefinitionsH

Anatomy of a VLA

Definition— Vision-language-action model

A VLA is a learned policy

πθ(at,at+1,…,at+H−1  ∣  ot,ℓ,st)\pi_\theta\big(a_{t}, a_{t+1}, \ldots, a_{t+H-1} \;\big|\; o_t, \ell, s_t\big)

that maps the current camera images oto_t, a natural-language instruction ℓ\ell and the robot's proprioceptive state sts_t (joint angles, gripper opening) to a distribution over the next HH actions, an action chunk. Its backbone is initialised from a pretrained vision-language model, and it is trained on robot demonstrations, usually together with ordinary vision-language data.

An action is whatever the low-level controller accepts: commonly a 7-number end-effector command (a change of position, a change of orientation, a gripper command) or a vector of joint-position targets; for a humanoid with two hands it can exceed 40 numbers. Predicting a chunk rather than a single step, an idea popularised by ACT, smooths motion and lets a slow model drive a fast controller: π0 predicts H=50H = 50 steps at a time and executes part of each chunk before predicting again.

camera imagesinstruction:"put the mug in the sink"joint statevision-languagebackbonebillions of parametersa few to ~10 Hzaction headtokens, or aflow expertaction chunka(t) … a(t+H−1)controller100 Hz – 1 kHzrobotnew observations
A VLA as most labs build it in 2026. A pretrained vision-language backbone reads images, the instruction and the robot's state; an action head turns its output into a chunk of future actions; a conventional controller tracks them at a much higher rate.

The figure hides a timing problem that shapes every design. A backbone with billions of parameters, on a GPU that fits inside a robot, produces a few forward passes per second; a dexterous arm wants new targets 50 times a second and its motor loops run at a kilohertz. Labs resolve this in three ways, often together: action chunks; a small, fast action module attached to the big model (π0's 300-million-parameter "action expert"); and an explicit split into a slow "System 2" that reasons and a fast "System 1" that moves. Figure's Helix (February 2025) runs a 7-billion-parameter VLM at 7–9 Hz and an 80-million-parameter visuomotor policy at 200 Hz over 35 degrees of freedom of a humanoid's upper body; NVIDIA's GR00T N1 (March 2025) pairs a VLM with a diffusion-transformer action module in the same spirit.

ComputationC

Actions as tokens

The simplest way to make a language model emit actions is to make actions look like words. RT-1 (Google, December 2022), a 35-million-parameter transformer trained on about 130,000 demonstrations collected over 17 months with 13 robots, discretised each action dimension into 256 bins and predicted one token per dimension. RT-2 (July 2023) applied the same trick to large VLMs (PaLI-X and PaLM-E, up to 55 billion parameters): action bins were mapped onto existing vocabulary tokens, so an action became a short "sentence" of integers the model could be fine-tuned to write. OpenVLA (June 2024), an open 7-billion-parameter model built on Llama 2 with DINOv2 and SigLIP vision encoders and trained on 970,000 episodes from the Open X-Embodiment collection, overwrote the 256 least-used tokens of its vocabulary with action bins.

Definition— Uniform action binning

Fix bounds [ℓj,uj][\ell_j, u_j] for action dimension jj and a number of bins NN (256 in all three models). The bin width is Δj=(uj−ℓj)/N\Delta_j = (u_j - \ell_j)/N. An action value aa is clipped to the bounds and mapped to the token k=⌊(a−ℓj)/Δj⌋k = \lfloor (a - \ell_j)/\Delta_j \rfloor (with a=uja = u_j sent to bin N−1N-1); a token is decoded to the centre of its bin, a^=ℓj+(k+12)Δj\hat a = \ell_j + (k + \tfrac12)\Delta_j.

Theorem— Quantisation error of uniform binning

For any action inside the bounds, ∣a^−a∣≤Δ/2|\hat a - a| \le \Delta/2. If actions are spread evenly across each bin, the error is uniform on [−Δ/2,Δ/2][-\Delta/2, \Delta/2] and its root-mean-square value is Δ/12\Delta/\sqrt{12}.

Proof

A value in bin kk lies in [ℓ+kΔ,ℓ+(k+1)Δ)[\ell + k\Delta, \ell + (k+1)\Delta), and the decoded centre is the midpoint of that interval, so it is at most half a width away. If the error ee is uniform on [−Δ/2,Δ/2][-\Delta/2, \Delta/2], its mean square is 1Δ∫−Δ/2Δ/2e2 de=Δ2/12\frac{1}{\Delta}\int_{-\Delta/2}^{\Delta/2} e^2\,de = \Delta^2/12.

∎

The theorem says nothing about values outside the bounds, and that is where the real decision lies. Here is one action dimension of a made-up but realistic dataset — a few millimetres of motion per step, plus ten teleoperation glitches — binned two ways.

python
import numpy as np

def tokenize(a, low, high, n_bins=256):
    """Map continuous actions to integer tokens 0..n_bins-1 (uniform bins on [low, high])."""
    a = np.clip(a, low, high)                                   # values outside the range are clipped
    idx = np.floor((a - low) / (high - low) * n_bins).astype(int)
    return np.minimum(idx, n_bins - 1)                          # a == high belongs to the last bin

def detokenize(tokens, low, high, n_bins=256):
    """Map tokens back to the centre of their bin."""
    return low + (tokens + 0.5) * (high - low) / n_bins

# One action dimension: the end effector's x-displacement per control step, in metres.
# Most steps move a few millimetres; a handful of teleoperation glitches jump by centimetres.
rng = np.random.default_rng(7)
dx = rng.normal(0.0, 0.004, size=20_000)
glitches = rng.choice(dx.size, size=10, replace=False)
dx[glitches] = rng.choice([-1, 1], size=10) * rng.uniform(0.05, 0.09, size=10)

for name, (low, high) in {
    "min/max of the data": (dx.min(), dx.max()),
    "1st-99th percentile": tuple(np.percentile(dx, [1, 99])),
}.items():
    width = (high - low) / 256
    err = detokenize(tokenize(dx, low, high), low, high) - dx
    inside = (dx >= low) & (dx <= high)
    print(f"bounds from the {name}: [{low * 1000:+.2f}, {high * 1000:+.2f}] mm, bin width {width * 1e6:.1f} um")
    print(f"   inside the bounds: max error {np.abs(err[inside]).max() * 1e6:6.1f} um (theory {width / 2 * 1e6:.1f}),"
          f" RMS {np.sqrt(np.mean(err[inside] ** 2)) * 1e6:5.1f} um (theory {width / np.sqrt(12) * 1e6:.1f})")
    if (~inside).any():
        print(f"   clipped: {np.sum(~inside)} of {dx.size} steps, median error {np.median(np.abs(err[~inside])) * 1000:.1f} mm,"
              f" worst {np.abs(err).max() * 1000:.1f} mm")
    else:
        print("   clipped: none")

# How many tokens does per-dimension, per-step tokenisation need?
for hz in [3, 6, 50]:
    print(f"7-D arm at {hz:2d} Hz: {7 * hz:3d} action tokens per second")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With bounds taken from the data's minimum and maximum, ten glitches out of 20,000 steps stretch the range to ±88 mm and every bin to 0.69 mm wide: the errors match the theorem exactly (maximum 344 µm, RMS 200 µm against a predicted 199), but the resolution is wasted on motions that never happen. Bounds at the 1st and 99th percentiles, which is what OpenVLA uses, make the bins ten times finer (72 µm) at the price of clipping 2% of steps — by a median of 1.1 mm, and by 80 mm for the glitches, which is arguably what you want. Either way, the quantisation error of a single step is far below what a position-controlled arm can repeat, so precision is not the main weakness of binning.

Compressing the chunk: FAST

There is a subtler problem too. At 50 Hz consecutive actions are nearly identical, so a model trained to predict the next token learns that "copy the previous token" is almost always right, and the training signal per token shrinks. Physical Intelligence's FAST tokeniser (January 2025) attacks both problems the way JPEG compresses images: take a whole chunk, apply a discrete cosine transform (DCT) to each action dimension, round the scaled coefficients to integers (most high-frequency ones become zero), then compress the integer sequence with byte-pair encoding, the tokeniser of Language models. The sketch below does the first three steps on a one-second chunk for a 7-joint arm.

python
import numpy as np

def dct_matrix(n):
    """Orthonormal DCT-II matrix: row k is a cosine with k half-periods over n samples."""
    k = np.arange(n)[:, None]
    i = np.arange(n)[None, :]
    D = np.sqrt(2 / n) * np.cos(np.pi * (i + 0.5) * k / n)
    D[0] /= np.sqrt(2)
    return D

# A one-second action chunk at 50 Hz for a 7-joint arm: smooth motions plus a little noise.
rng = np.random.default_rng(3)
H, dims = 50, 7
t = np.linspace(0, 1, H)
chunk = np.stack([0.3 * np.sin(2 * np.pi * (0.4 + 0.1 * j) * t + j) + 0.1 * t for j in range(dims)], axis=1)
chunk += rng.normal(0, 0.0005, size=chunk.shape)                # joint targets in radians

D = dct_matrix(H)
coeffs = D @ chunk                                              # 50 frequency coefficients per joint
print("DCT matrix orthonormal:", np.allclose(D @ D.T, np.eye(H)))
share = np.sort((coeffs ** 2).ravel())[::-1].cumsum() / np.sum(coeffs ** 2)
print(f"energy in the largest 20 of {H * dims} coefficients: {share[19]:.1%}")

# Per-step binning for comparison: 256 bins per joint over the chunk's range.
low, high = chunk.min(), chunk.max()
width = (high - low) / 256
binned = low + (np.minimum(np.floor((chunk - low) / width), 255) + 0.5) * width
print(f"\nper-step binning: {H * dims} tokens, RMS error {np.sqrt(np.mean((binned - chunk) ** 2)) * 1000:.2f} mrad")
print(f"range of the chunk: {(high - low) * 1000:.0f} mrad")

print("\nDCT + rounding:  scale   nonzero integers   RMS error (mrad)")
for scale in [10, 30, 100, 300]:
    q = np.round(coeffs * scale)                                # most high-frequency coefficients round to 0
    recon = D.T @ (q / scale)                                   # the inverse DCT is the transpose
    rms = np.sqrt(np.mean((recon - chunk) ** 2)) * 1000
    print(f"{scale:22d}   {int(np.count_nonzero(q)):8d} of {H * dims}   {rms:12.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Twenty of 350 coefficients carry 97.8% of the motion's energy. At a coarse rounding scale the chunk becomes 35 nonzero integers with an RMS error of 12 mrad (under 2% of the 665 mrad range of motion); finer scales buy precision with more nonzero integers, and at 154 integers the error has fallen to 0.85 mrad, about what per-step binning achieves with 350 tokens. The trade is tunable, and byte-pair encoding then merges the long runs of zeros into a few tokens. In the FAST paper, π0 trained with these tokens ("π0-FAST") matched the diffusion version of π0 while training up to five times faster, though autoregressive decoding still made it slower at run time.

ComputationC

Continuous action heads: diffusion and flow matching

The other family never discretises at all. Its starting point is a failure you can see in one picture.

startgoalpostdemonstrations going leftdemonstrations going rightaverage action
Demonstrations of reaching past a post, seen from above. Half the operators went left, half went right. A policy trained to minimise squared error predicts their average — straight into the post.

Human demonstrations are multimodal: there are several good ways to do most things. A network trained with squared error learns the conditional mean, which for two modes can be the one action nobody took. Token outputs escape this because a categorical distribution can put mass on two separate bins. Continuous heads escape it by being generative models: Diffusion Policy (Columbia, MIT and Toyota Research Institute, 2023) learns to denoise random action sequences into demonstrated ones, and flow matching, used by π0, GR00T and many 2025–26 models, learns a velocity field that carries noise to actions.

Definition— Flow matching (linear path)

Draw noise x0∼N(0,I)x_0 \sim \mathcal{N}(0, I) and a demonstrated action chunk x1x_1, and define xt=(1−t) x0+t x1x_t = (1-t)\,x_0 + t\,x_1 for t∈[0,1]t \in [0, 1]. A network vθ(x,t∣context)v_\theta(x, t \mid \text{context}) is trained by regression on the straight-line velocity:

L(θ)=Et, x0, x1∥vθ(xt,t)−(x1−x0)∥2.\mathcal{L}(\theta) = \mathbb{E}_{t,\,x_0,\,x_1} \big\| v_\theta(x_t, t) - (x_1 - x_0) \big\|^2 .

The minimiser is the marginal velocity field v∗(x,t)=E[ x1−x0∣xt=x ]=(E[x1∣xt=x]−x)/(1−t)v^\ast(x, t) = \mathbb{E}[\,x_1 - x_0 \mid x_t = x\,] = \big(\mathbb{E}[x_1 \mid x_t = x] - x\big)/(1-t). To act, sample fresh noise and integrate x˙=vθ(x,t)\dot x = v_\theta(x, t) from t=0t = 0 to t=1t = 1, for instance with a few Euler steps.

Notice that the loss is a squared error, yet the result is not an average: the network regresses a velocity at every intermediate point, and following those velocities sorts each noise sample into one mode. For a mixture of Gaussians, v∗v^\ast has a closed form, so we can watch the ideal, perfectly trained head at work without training anything. The code compares three output heads on the post example, with sideways offsets of ±15 cm and a post 12 cm wide.

python
import numpy as np

# Demonstrations of reaching past a post: the hand passes left (-15 cm) or right (+15 cm) of it,
# half the time each. We look at one action number: the sideways offset y as the hand passes the post.
rng = np.random.default_rng(11)
modes_m, spread_m = np.array([-0.15, 0.15]), 0.02       # metres
demos = rng.choice(modes_m, size=2000) + rng.normal(0, spread_m, size=2000)
post = 0.06                                             # anything with |y| < 6 cm hits the post

# 1. Regression with squared error predicts the average action.
print(f"regression (MSE) prediction: y = {demos.mean() * 100:+.1f} cm -> hits the post: {abs(demos.mean()) < post}")

# 2. Tokens: a categorical distribution over 256 bins can hold both modes.
edges = np.linspace(-0.3, 0.3, 257)
probs = np.histogram(demos, bins=edges)[0] / demos.size
centres = (edges[:-1] + edges[1:]) / 2
print(f"token distribution: P(a bin inside the post) = {probs[np.abs(centres) < post].sum():.4f}")

# 3. Flow matching, in normalised units (actions divided by their standard deviation, as VLAs do).
#    Start from noise x ~ N(0, 1) at t = 0 and follow a velocity field to t = 1. For a mixture of
#    Gaussians the ideal field (what a perfectly trained network would learn) has a closed form:
#    v(x, t) = (E[action | x_t = x] - x) / (1 - t).
scale = demos.std()
modes, spread = modes_m / scale, spread_m / scale

def velocity(x, t):
    var = (1 - t) ** 2 + (t * spread) ** 2                         # variance of x_t given each mode
    logw = -0.5 * (x[:, None] - t * modes) ** 2 / var              # the two modes are equally likely
    w = np.exp(logw - logw.max(axis=1, keepdims=True))
    w /= w.sum(axis=1, keepdims=True)                              # P(mode | x_t = x)
    mean_given_mode = modes + t * spread ** 2 / var * (x[:, None] - t * modes)
    return ((w * mean_given_mode).sum(axis=1) - x) / (1 - t)

def sample(n, steps):
    x = rng.normal(size=n)
    for i in range(steps):                                         # Euler steps; pi0 uses 10
        x = x + velocity(x, i / steps) / steps
    return x * scale                                               # back to metres

print("\nflow matching, 4000 samples each:")
print(" Euler steps   hit the post   went left   spread of the left mode (cm)")
for steps in [1, 2, 3, 5, 10, 50]:
    y = sample(4000, steps)
    left = f"{np.std(y[y < 0]) * 100:10.1f}" if np.any(y < 0) else "   (all samples at the mean)"
    print(f"{steps:12d}   {np.mean(np.abs(y) < post):10.1%}   {np.mean(y < 0):9.2f}   {left}")
print(f"demonstrations: spread of the left mode {np.std(demos[demos < 0]) * 100:.1f} cm")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The regression head predicts −0.1 cm and drives into the post. The token distribution puts no mass inside it. The flow is the interesting one. With a single Euler step every sample lands on the mean, exactly like regression: at t=0t = 0 the field knows nothing about which mode a sample will join, so it points everywhere at the average. With two steps a third of samples still hit the post, with five steps 2%, and with ten — the number π0 uses — 0.1%, with each mode about as wide as in the demonstrations (1.7 cm against 2.1). Integration steps are the knob that trades speed for faithfulness, which is why distilled and "consistency" variants that sample in one or two steps are an active topic.

Token heads have their own version of the trap. If each action dimension's token is sampled independently, a robot can draw "left" for its xx and "right" for its yy and produce a combination no demonstrator made. Autoregressive decoding, where each token is conditioned on the ones before it, avoids this, at the price of decoding them one by one.

ExplanationH

Co-training on web and robot data

Fine-tune a VLM only on robot data and it forgets: within a few thousand steps it stops answering questions about images and, it turns out, gets worse at following unfamiliar instructions on the robot too. RT-2's remedy, co-fine-tuning, was to keep mixing the original web vision-language tasks (captioning, visual question answering) into every training batch alongside robot trajectories. The payoff was the paper's headline: on scenes, objects and backgrounds absent from the robot data, success rose from RT-1's 32% to 62%, and the robot could follow commands that needed web knowledge, such as moving a can onto a picture of a named celebrity, or choosing the object that could serve as an improvised hammer (a rock).

Later recipes refine the mixture:

  • π0.5 (Physical Intelligence, April 2025) co-trains on data from many robots, web data (captions, question answering, object detection), human-annotated high-level subtasks ("pick up the pillow") and verbal instructions. At run time the same model first predicts a subtask in text, then the actions for it. It was the first end-to-end system shown cleaning kitchens and bedrooms in homes it had never seen, and the paper's ablations found that removing the web data mainly hurt the robot's handling of object categories absent from its robot data.
  • Knowledge insulation (Physical Intelligence, May 2025) addresses a quieter problem: gradients from a freshly initialised flow-matching action expert flow back into the VLM and damage what it knew. The fix is to stop those gradients, train the backbone on FAST tokens (cheap, language-like), and train the action expert on top: "train fast, run fast, generalise better".
  • GR00T N1 describes a "data pyramid": web data and human videos at the base, synthetic data from simulation and video-generation models in the middle, real robot trajectories at the top. For videos with no recorded actions it learns latent actions, codes for "what changed between these frames" that can be trained on like real actions.
  • Gemini Robotics 1.5 (Google DeepMind, October 2025) adds "motion transfer", training one model on very different robots — a bimanual ALOHA, a two-arm Franka setup and Apptronik's Apollo humanoid — so that skills learned on one appear on another, and lets the model write out a short chain of reasoning in language before it acts.

The mixture weights, the order of training stages and which parameters are frozen are among the most closely guarded details in the field; papers report them unevenly.

ExplanationH

The models of 2023–2026, from their papers

The table lists representative models, with what their authors report. It is not a ranking: they were evaluated on different robots and tasks, and few were compared head to head.

Model (organisation, date)Backbone and sizeAction outputOpenness
RT-2 (Google DeepMind, US/UK, 2023)PaLI-X or PaLM-E, up to 55B256-bin tokensclosed
Octo (UC Berkeley, Stanford, CMU and others, 2024)transformer trained from scratch, 27M–93Mdiffusion headopen
OpenVLA (Stanford, UC Berkeley, TRI and others, 2024)Llama 2 + DINOv2/SigLIP, 7B256-bin tokens; OFT variant continuousopen
RDT-1B (Tsinghua University, China, 2024)diffusion transformer, 1.2Bdiffusionopen
π0, π0-FAST, π0.5 (Physical Intelligence, US, 2024–25)PaliGemma 3B + action expert, 3.3Bflow matching, or FAST tokensopen (openpi)
Helix (Figure, US, 2025)7B VLM + 80M visuomotor policycontinuous, 200 Hzclosed
GR00T N1 → N1.7 (NVIDIA, US, 2025–26)VLM + diffusion transformer, 2–3Bflow matchingopen; N1.7 under Apache 2.0
Gemini Robotics 1.0 → 2 (Google DeepMind, UK/US, 2025–26)Gemini models, cloud and on-device versionsnot fully disclosedclosed; reasoning model by API, VLA for partners
GO-1 (AgiBot, China, 2025)VLM + latent action plannercontinuousreleased with the AgiBot World dataset
GR-3 (ByteDance Seed, China, 2025)VLA co-trained with web data and VR human datacontinuoustechnical report
SmolVLA (Hugging Face, France/US, 2025)small VLM, 0.45Bflow matchingopen

Three lines of development stand out in 2025–26. Learning from deployment: Physical Intelligence's π*0.6 (November 2025) adds reinforcement learning from the robot's own attempts and from human corrections, and reports roughly halving failure rates on its hardest tasks — making espresso, assembling boxes, folding laundry in real homes. Steerability: π0.7 (April 2026) is conditioned not only on an instruction but on metadata about how to do the task and on subgoal images from a small world model, which lets it train on mixed-quality data, including failures. Whole bodies: Gemini Robotics 2 (July 2026) controls a walking, crouching Apollo humanoid with five-fingered hands, and NVIDIA's GR00T N1.6 and N1.7 target full humanoid bodies. Lumenary Robotics' next lessons take up humanoids and the world models now feeding these policies.

Working with a VLA in practice

Most teams do not pretrain a VLA; they fine-tune an open one on 50–500 demonstrations of their own task, often with low-rank adapters, using openpi, Isaac GR00T or Hugging Face's LeRobot, and serve it from a GPU on or next to the robot. The control loop is short. The sketch below is schematic Python (it needs a real model, cameras and a robot, so it is not run here; each library has its own interface):

python
# Schematic only: not run here. See openpi, Isaac-GR00T or LeRobot for real interfaces.
while not task_done:
    obs = {"images": cameras.read(), "state": robot.joint_positions(), "prompt": "fold the towel"}
    chunk = policy.infer(obs)          # e.g. 50 future actions, normalised then un-normalised
    for action in chunk[:25]:          # execute part of the chunk, then re-plan with fresh images
        robot.command(action)          # a conventional controller tracks each target at high rate
        safety_monitor.check(robot)    # limits, collisions, emergency stop: see the last lesson

Executing only part of a chunk keeps the policy reactive; Physical Intelligence's "real-time chunking" (2025) goes further, computing the next chunk while the current one runs and constraining it to agree with the actions already committed, so the arm never pauses. Evaluation, safety and deployment covers the monitor in the last line.

ExplanationH

What generalises, and how VLAs fail (as of late 2026)

What works. Picking and placing everyday objects never seen in training, in new rooms, from plain instructions, is now routine for the best models. Long, multi-step household tasks in unfamiliar homes work some of the time (π0.5's cleaning; π0.7's appliance tasks). Deformable and dexterous skills — folding laundry, assembling boxes, making coffee — work well after task-specific fine-tuning or reinforcement learning. Transfer between robot bodies is real but partial: π0.7 reports folding laundry zero-shot on a bimanual UR5e setup it had no training data for.

How good, in numbers. Few labs publish success rates against a fixed protocol, and those that do deserve credit. OpenVLA reported beating the 55-billion-parameter RT-2-X by 16.5 percentage points across 29 tasks with seven times fewer parameters. Google DeepMind's Gemini Robotics 2 page reports, for its Apollo humanoid with five-fingered hands, 92% on unscrewing a light bulb but 36% on screwing one in, 44% on tying a bin bag, and 45.7% on picking objects up from the floor. Those are frontier results in July 2026, and they are not the 99.9% a factory needs.

How they fail.

  • Ignoring the instruction. Perturbation studies of models that score above 90% on the LIBERO simulation benchmark found that performance can collapse when objects, layouts or camera angles change, and that the models often execute the same motion whatever the instruction says, even when it is replaced by nonsense (LIBERO-PRO and LIBERO-Plus, October 2025). High benchmark scores can reflect memorised trajectories.
  • Sensitivity to viewpoint and set-up. A camera moved by a few centimetres, a different table height or new lighting can cut success sharply; LIBERO-Plus measured drops from 95% to below 30%.
  • Force and precision. Most VLAs output position targets from images and have no sense of touch, so tight insertions and tasks that depend on contact force remain hard (Dexterous hands and touch).
  • Recovery and memory. Policies conditioned on the current frame forget what they have already done; after a slip they may repeat a step or give up. Long tasks compound small error rates.
  • Not knowing when they are failing. A VLA's outputs carry no reliable confidence, so a robot can continue confidently in a situation it has never seen. Runtime monitors and safety filters exist for exactly this reason.
  • Evaluation itself. Real-robot trials are slow and noisy, and many published comparisons rest on 10–20 trials per task: the statistics of the last lesson of this course matter here.
Common misconceptionsH

Common misconceptions

Misconception

"A VLA understands physics because it understands language." The backbone knows what objects are called and roughly where they are. How things move under force comes only from robot data (and, increasingly, from video), and that is the scarcest data in the field.

Misconception

"Bigger backbones make better robots." Size helps semantic generalisation, but OpenVLA (7B) beat RT-2-X (55B), and SmolVLA (0.45B) is competitive on many benchmarks. Data diversity, action representation and control rate matter at least as much, and a bigger model is slower at the one thing a robot cannot wait for.

Misconception

"Tokenised actions are too coarse for precise work." With 256 bins over sensible bounds the quantisation error is tens of micrometres per step, below an arm's repeatability. The real costs of tokens are decoding speed at high control rates and, for naive schemes, weak training signals — which is what FAST and continuous heads address.

Misconception

"The demo video shows what the robot can do." Videos are selected, sometimes sped up, and occasionally teleoperated; responsible labs say which. A success rate over a stated number of trials, under a stated protocol, is the claim to look for.

Misconception

"Flow matching is regression, so it averages modes too." It regresses a velocity at every intermediate noise level, and integrating that field separates the modes. Only a one-step sampler collapses to the mean, as the simulation showed.

Exercises

Exercises

1. Vision-language-action models

Warm-upHuman

A VLA tokenises the wrist's sideways displacement per step with 256256 uniform bins over [−20,+20][-20, +20] mm, decoding each token to the centre of its bin. For an action inside the bounds, what is the largest possible quantisation error, in micrometres (rounded to the nearest whole number)?

2. Vision-language-action models

Warm-upHuman

An autoregressive VLA on a robot's onboard GPU decodes 140140 tokens per second. It uses one token per action dimension per control step (RT-2 and OpenVLA style) and produces one step at a time. What is the highest control rate, in Hz, it can sustain for a 7-dimensional arm?

3. Vision-language-action models

CoreHuman

A robot learns to set a cup down on a shelf from 400 demonstrations. Half the operators placed it in the left-hand gap between two jars and half in the right-hand gap; the middle is occupied by a jar. Which output head is most likely to knock the jar over?

Choose one

4. Vision-language-action models

CoreHuman

In the lesson's simulation, a flow-matching head sampled with a single Euler step put every sample exactly at the mean of the demonstrations. Why?

Choose one

5. Vision-language-action models

CoreHuman

Which of these are documented weaknesses of vision-language-action models as of 2026? Select all that apply.

Choose all that apply

6. Vision-language-action models

CoreComputational

Implement OpenVLA-style action binning for a whole dataset. fit_bounds(actions) returns per-dimension bounds at the 1st and 99th percentiles of each column of an (n, d) array. tokenize maps actions to integer tokens in 0 .. n_bins-1 (clipping values outside the bounds, and sending a value equal to high to the last bin). detokenize maps tokens back to the centres of their bins.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Vision-language-action models

CoreComputational

A flow-matching action head is sampled by integrating x˙=v(x,t)\dot x = v(x, t) from t=0t = 0 to t=1t = 1. Write integrate(velocity, x0, steps), which takes steps equal Euler steps, evaluating velocity(x, t) at t=0,1steps,…,1−1stepst = 0, \tfrac{1}{\text{steps}}, \ldots, 1 - \tfrac{1}{\text{steps}}, and returns xx at t=1t = 1. (It must never evaluate the field at t=1t = 1, where the ideal field divides by 1−t1 - t.)

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources