From describing a scene to acting in it
Show a modern vision-language model (VLM) a photograph of a kitchen counter and ask what is there, and it will tell you: a chipped blue mug, a sponge, a bottle of washing-up liquid, and that the mug probably belongs in the sink. Ask it which way to move a robot's wrist, in metres, for the next tenth of a second, and it has no idea. Its knowledge came from billions of images and pages of text, and none of them contained a joint angle.
The policies of Imitation learning have the opposite problem. Trained on a few hundred demonstrations of one task, a diffusion policy can fold a towel with real finesse, but swap the towel for a T-shirt, or ask it in words to do something else, and it is lost. It knows how to move and almost nothing about the world.
A vision-language-action model (VLA) tries to have both. Start from a VLM pretrained on the web, keep training it on robot demonstrations until it outputs motor commands, and hope the semantic knowledge carries over: that a robot that has never been taught the word "dinosaur" can still pick up the toy dinosaur, because the VLM knew what one looks like. The bet was made publicly by Google's RT-2 in July 2023, and it paid off well enough that by 2026 every major robot-learning lab, and most well-funded robot companies, trains one.
This lesson takes the VLA apart. We will look at its anatomy, implement the two main ways of producing actions — discrete tokens and continuous flow-matching heads — measure what each costs, then turn to how these models are trained and what, honestly, they can and cannot do as of late 2026. It assumes the transformer of Attention and the transformer and the pretraining story of Language models, and the datasets of Robot data at scale.
Anatomy of a VLA
Definition— Vision-language-action model
A VLA is a learned policy
that maps the current camera images , a natural-language instruction and the robot's proprioceptive state (joint angles, gripper opening) to a distribution over the next actions, an action chunk. Its backbone is initialised from a pretrained vision-language model, and it is trained on robot demonstrations, usually together with ordinary vision-language data.
An action is whatever the low-level controller accepts: commonly a 7-number end-effector command (a change of position, a change of orientation, a gripper command) or a vector of joint-position targets; for a humanoid with two hands it can exceed 40 numbers. Predicting a chunk rather than a single step, an idea popularised by ACT, smooths motion and lets a slow model drive a fast controller: π0 predicts steps at a time and executes part of each chunk before predicting again.
The figure hides a timing problem that shapes every design. A backbone with billions of parameters, on a GPU that fits inside a robot, produces a few forward passes per second; a dexterous arm wants new targets 50 times a second and its motor loops run at a kilohertz. Labs resolve this in three ways, often together: action chunks; a small, fast action module attached to the big model (π0's 300-million-parameter "action expert"); and an explicit split into a slow "System 2" that reasons and a fast "System 1" that moves. Figure's Helix (February 2025) runs a 7-billion-parameter VLM at 7–9 Hz and an 80-million-parameter visuomotor policy at 200 Hz over 35 degrees of freedom of a humanoid's upper body; NVIDIA's GR00T N1 (March 2025) pairs a VLM with a diffusion-transformer action module in the same spirit.
Actions as tokens
The simplest way to make a language model emit actions is to make actions look like words. RT-1 (Google, December 2022), a 35-million-parameter transformer trained on about 130,000 demonstrations collected over 17 months with 13 robots, discretised each action dimension into 256 bins and predicted one token per dimension. RT-2 (July 2023) applied the same trick to large VLMs (PaLI-X and PaLM-E, up to 55 billion parameters): action bins were mapped onto existing vocabulary tokens, so an action became a short "sentence" of integers the model could be fine-tuned to write. OpenVLA (June 2024), an open 7-billion-parameter model built on Llama 2 with DINOv2 and SigLIP vision encoders and trained on 970,000 episodes from the Open X-Embodiment collection, overwrote the 256 least-used tokens of its vocabulary with action bins.
Definition— Uniform action binning
Fix bounds for action dimension and a number of bins (256 in all three models). The bin width is . An action value is clipped to the bounds and mapped to the token (with sent to bin ); a token is decoded to the centre of its bin, .
Theorem— Quantisation error of uniform binning
For any action inside the bounds, . If actions are spread evenly across each bin, the error is uniform on and its root-mean-square value is .
Proof
A value in bin lies in , and the decoded centre is the midpoint of that interval, so it is at most half a width away. If the error is uniform on , its mean square is .
∎
The theorem says nothing about values outside the bounds, and that is where the real decision lies. Here is one action dimension of a made-up but realistic dataset — a few millimetres of motion per step, plus ten teleoperation glitches — binned two ways.
import numpy as np
def tokenize(a, low, high, n_bins=256):
"""Map continuous actions to integer tokens 0..n_bins-1 (uniform bins on [low, high])."""
a = np.clip(a, low, high) # values outside the range are clipped
idx = np.floor((a - low) / (high - low) * n_bins).astype(int)
return np.minimum(idx, n_bins - 1) # a == high belongs to the last bin
def detokenize(tokens, low, high, n_bins=256):
"""Map tokens back to the centre of their bin."""
return low + (tokens + 0.5) * (high - low) / n_bins
# One action dimension: the end effector's x-displacement per control step, in metres.
# Most steps move a few millimetres; a handful of teleoperation glitches jump by centimetres.
rng = np.random.default_rng(7)
dx = rng.normal(0.0, 0.004, size=20_000)
glitches = rng.choice(dx.size, size=10, replace=False)
dx[glitches] = rng.choice([-1, 1], size=10) * rng.uniform(0.05, 0.09, size=10)
for name, (low, high) in {
"min/max of the data": (dx.min(), dx.max()),
"1st-99th percentile": tuple(np.percentile(dx, [1, 99])),
}.items():
width = (high - low) / 256
err = detokenize(tokenize(dx, low, high), low, high) - dx
inside = (dx >= low) & (dx <= high)
print(f"bounds from the {name}: [{low * 1000:+.2f}, {high * 1000:+.2f}] mm, bin width {width * 1e6:.1f} um")
print(f" inside the bounds: max error {np.abs(err[inside]).max() * 1e6:6.1f} um (theory {width / 2 * 1e6:.1f}),"
f" RMS {np.sqrt(np.mean(err[inside] ** 2)) * 1e6:5.1f} um (theory {width / np.sqrt(12) * 1e6:.1f})")
if (~inside).any():
print(f" clipped: {np.sum(~inside)} of {dx.size} steps, median error {np.median(np.abs(err[~inside])) * 1000:.1f} mm,"
f" worst {np.abs(err).max() * 1000:.1f} mm")
else:
print(" clipped: none")
# How many tokens does per-dimension, per-step tokenisation need?
for hz in [3, 6, 50]:
print(f"7-D arm at {hz:2d} Hz: {7 * hz:3d} action tokens per second")With bounds taken from the data's minimum and maximum, ten glitches out of 20,000 steps stretch the range to ±88 mm and every bin to 0.69 mm wide: the errors match the theorem exactly (maximum 344 µm, RMS 200 µm against a predicted 199), but the resolution is wasted on motions that never happen. Bounds at the 1st and 99th percentiles, which is what OpenVLA uses, make the bins ten times finer (72 µm) at the price of clipping 2% of steps — by a median of 1.1 mm, and by 80 mm for the glitches, which is arguably what you want. Either way, the quantisation error of a single step is far below what a position-controlled arm can repeat, so precision is not the main weakness of binning.
Compressing the chunk: FAST
There is a subtler problem too. At 50 Hz consecutive actions are nearly identical, so a model trained to predict the next token learns that "copy the previous token" is almost always right, and the training signal per token shrinks. Physical Intelligence's FAST tokeniser (January 2025) attacks both problems the way JPEG compresses images: take a whole chunk, apply a discrete cosine transform (DCT) to each action dimension, round the scaled coefficients to integers (most high-frequency ones become zero), then compress the integer sequence with byte-pair encoding, the tokeniser of Language models. The sketch below does the first three steps on a one-second chunk for a 7-joint arm.
import numpy as np
def dct_matrix(n):
"""Orthonormal DCT-II matrix: row k is a cosine with k half-periods over n samples."""
k = np.arange(n)[:, None]
i = np.arange(n)[None, :]
D = np.sqrt(2 / n) * np.cos(np.pi * (i + 0.5) * k / n)
D[0] /= np.sqrt(2)
return D
# A one-second action chunk at 50 Hz for a 7-joint arm: smooth motions plus a little noise.
rng = np.random.default_rng(3)
H, dims = 50, 7
t = np.linspace(0, 1, H)
chunk = np.stack([0.3 * np.sin(2 * np.pi * (0.4 + 0.1 * j) * t + j) + 0.1 * t for j in range(dims)], axis=1)
chunk += rng.normal(0, 0.0005, size=chunk.shape) # joint targets in radians
D = dct_matrix(H)
coeffs = D @ chunk # 50 frequency coefficients per joint
print("DCT matrix orthonormal:", np.allclose(D @ D.T, np.eye(H)))
share = np.sort((coeffs ** 2).ravel())[::-1].cumsum() / np.sum(coeffs ** 2)
print(f"energy in the largest 20 of {H * dims} coefficients: {share[19]:.1%}")
# Per-step binning for comparison: 256 bins per joint over the chunk's range.
low, high = chunk.min(), chunk.max()
width = (high - low) / 256
binned = low + (np.minimum(np.floor((chunk - low) / width), 255) + 0.5) * width
print(f"\nper-step binning: {H * dims} tokens, RMS error {np.sqrt(np.mean((binned - chunk) ** 2)) * 1000:.2f} mrad")
print(f"range of the chunk: {(high - low) * 1000:.0f} mrad")
print("\nDCT + rounding: scale nonzero integers RMS error (mrad)")
for scale in [10, 30, 100, 300]:
q = np.round(coeffs * scale) # most high-frequency coefficients round to 0
recon = D.T @ (q / scale) # the inverse DCT is the transpose
rms = np.sqrt(np.mean((recon - chunk) ** 2)) * 1000
print(f"{scale:22d} {int(np.count_nonzero(q)):8d} of {H * dims} {rms:12.2f}")Twenty of 350 coefficients carry 97.8% of the motion's energy. At a coarse rounding scale the chunk becomes 35 nonzero integers with an RMS error of 12 mrad (under 2% of the 665 mrad range of motion); finer scales buy precision with more nonzero integers, and at 154 integers the error has fallen to 0.85 mrad, about what per-step binning achieves with 350 tokens. The trade is tunable, and byte-pair encoding then merges the long runs of zeros into a few tokens. In the FAST paper, π0 trained with these tokens ("π0-FAST") matched the diffusion version of π0 while training up to five times faster, though autoregressive decoding still made it slower at run time.
Continuous action heads: diffusion and flow matching
The other family never discretises at all. Its starting point is a failure you can see in one picture.
Human demonstrations are multimodal: there are several good ways to do most things. A network trained with squared error learns the conditional mean, which for two modes can be the one action nobody took. Token outputs escape this because a categorical distribution can put mass on two separate bins. Continuous heads escape it by being generative models: Diffusion Policy (Columbia, MIT and Toyota Research Institute, 2023) learns to denoise random action sequences into demonstrated ones, and flow matching, used by π0, GR00T and many 2025–26 models, learns a velocity field that carries noise to actions.
Definition— Flow matching (linear path)
Draw noise and a demonstrated action chunk , and define for . A network is trained by regression on the straight-line velocity:
The minimiser is the marginal velocity field . To act, sample fresh noise and integrate from to , for instance with a few Euler steps.
Notice that the loss is a squared error, yet the result is not an average: the network regresses a velocity at every intermediate point, and following those velocities sorts each noise sample into one mode. For a mixture of Gaussians, has a closed form, so we can watch the ideal, perfectly trained head at work without training anything. The code compares three output heads on the post example, with sideways offsets of ±15 cm and a post 12 cm wide.
import numpy as np
# Demonstrations of reaching past a post: the hand passes left (-15 cm) or right (+15 cm) of it,
# half the time each. We look at one action number: the sideways offset y as the hand passes the post.
rng = np.random.default_rng(11)
modes_m, spread_m = np.array([-0.15, 0.15]), 0.02 # metres
demos = rng.choice(modes_m, size=2000) + rng.normal(0, spread_m, size=2000)
post = 0.06 # anything with |y| < 6 cm hits the post
# 1. Regression with squared error predicts the average action.
print(f"regression (MSE) prediction: y = {demos.mean() * 100:+.1f} cm -> hits the post: {abs(demos.mean()) < post}")
# 2. Tokens: a categorical distribution over 256 bins can hold both modes.
edges = np.linspace(-0.3, 0.3, 257)
probs = np.histogram(demos, bins=edges)[0] / demos.size
centres = (edges[:-1] + edges[1:]) / 2
print(f"token distribution: P(a bin inside the post) = {probs[np.abs(centres) < post].sum():.4f}")
# 3. Flow matching, in normalised units (actions divided by their standard deviation, as VLAs do).
# Start from noise x ~ N(0, 1) at t = 0 and follow a velocity field to t = 1. For a mixture of
# Gaussians the ideal field (what a perfectly trained network would learn) has a closed form:
# v(x, t) = (E[action | x_t = x] - x) / (1 - t).
scale = demos.std()
modes, spread = modes_m / scale, spread_m / scale
def velocity(x, t):
var = (1 - t) ** 2 + (t * spread) ** 2 # variance of x_t given each mode
logw = -0.5 * (x[:, None] - t * modes) ** 2 / var # the two modes are equally likely
w = np.exp(logw - logw.max(axis=1, keepdims=True))
w /= w.sum(axis=1, keepdims=True) # P(mode | x_t = x)
mean_given_mode = modes + t * spread ** 2 / var * (x[:, None] - t * modes)
return ((w * mean_given_mode).sum(axis=1) - x) / (1 - t)
def sample(n, steps):
x = rng.normal(size=n)
for i in range(steps): # Euler steps; pi0 uses 10
x = x + velocity(x, i / steps) / steps
return x * scale # back to metres
print("\nflow matching, 4000 samples each:")
print(" Euler steps hit the post went left spread of the left mode (cm)")
for steps in [1, 2, 3, 5, 10, 50]:
y = sample(4000, steps)
left = f"{np.std(y[y < 0]) * 100:10.1f}" if np.any(y < 0) else " (all samples at the mean)"
print(f"{steps:12d} {np.mean(np.abs(y) < post):10.1%} {np.mean(y < 0):9.2f} {left}")
print(f"demonstrations: spread of the left mode {np.std(demos[demos < 0]) * 100:.1f} cm")The regression head predicts −0.1 cm and drives into the post. The token distribution puts no mass inside it. The flow is the interesting one. With a single Euler step every sample lands on the mean, exactly like regression: at the field knows nothing about which mode a sample will join, so it points everywhere at the average. With two steps a third of samples still hit the post, with five steps 2%, and with ten — the number π0 uses — 0.1%, with each mode about as wide as in the demonstrations (1.7 cm against 2.1). Integration steps are the knob that trades speed for faithfulness, which is why distilled and "consistency" variants that sample in one or two steps are an active topic.
Token heads have their own version of the trap. If each action dimension's token is sampled independently, a robot can draw "left" for its and "right" for its and produce a combination no demonstrator made. Autoregressive decoding, where each token is conditioned on the ones before it, avoids this, at the price of decoding them one by one.
Co-training on web and robot data
Fine-tune a VLM only on robot data and it forgets: within a few thousand steps it stops answering questions about images and, it turns out, gets worse at following unfamiliar instructions on the robot too. RT-2's remedy, co-fine-tuning, was to keep mixing the original web vision-language tasks (captioning, visual question answering) into every training batch alongside robot trajectories. The payoff was the paper's headline: on scenes, objects and backgrounds absent from the robot data, success rose from RT-1's 32% to 62%, and the robot could follow commands that needed web knowledge, such as moving a can onto a picture of a named celebrity, or choosing the object that could serve as an improvised hammer (a rock).
Later recipes refine the mixture:
- π0.5 (Physical Intelligence, April 2025) co-trains on data from many robots, web data (captions, question answering, object detection), human-annotated high-level subtasks ("pick up the pillow") and verbal instructions. At run time the same model first predicts a subtask in text, then the actions for it. It was the first end-to-end system shown cleaning kitchens and bedrooms in homes it had never seen, and the paper's ablations found that removing the web data mainly hurt the robot's handling of object categories absent from its robot data.
- Knowledge insulation (Physical Intelligence, May 2025) addresses a quieter problem: gradients from a freshly initialised flow-matching action expert flow back into the VLM and damage what it knew. The fix is to stop those gradients, train the backbone on FAST tokens (cheap, language-like), and train the action expert on top: "train fast, run fast, generalise better".
- GR00T N1 describes a "data pyramid": web data and human videos at the base, synthetic data from simulation and video-generation models in the middle, real robot trajectories at the top. For videos with no recorded actions it learns latent actions, codes for "what changed between these frames" that can be trained on like real actions.
- Gemini Robotics 1.5 (Google DeepMind, October 2025) adds "motion transfer", training one model on very different robots — a bimanual ALOHA, a two-arm Franka setup and Apptronik's Apollo humanoid — so that skills learned on one appear on another, and lets the model write out a short chain of reasoning in language before it acts.
The mixture weights, the order of training stages and which parameters are frozen are among the most closely guarded details in the field; papers report them unevenly.
The models of 2023–2026, from their papers
The table lists representative models, with what their authors report. It is not a ranking: they were evaluated on different robots and tasks, and few were compared head to head.
| Model (organisation, date) | Backbone and size | Action output | Openness |
|---|---|---|---|
| RT-2 (Google DeepMind, US/UK, 2023) | PaLI-X or PaLM-E, up to 55B | 256-bin tokens | closed |
| Octo (UC Berkeley, Stanford, CMU and others, 2024) | transformer trained from scratch, 27M–93M | diffusion head | open |
| OpenVLA (Stanford, UC Berkeley, TRI and others, 2024) | Llama 2 + DINOv2/SigLIP, 7B | 256-bin tokens; OFT variant continuous | open |
| RDT-1B (Tsinghua University, China, 2024) | diffusion transformer, 1.2B | diffusion | open |
| π0, π0-FAST, π0.5 (Physical Intelligence, US, 2024–25) | PaliGemma 3B + action expert, 3.3B | flow matching, or FAST tokens | open (openpi) |
| Helix (Figure, US, 2025) | 7B VLM + 80M visuomotor policy | continuous, 200 Hz | closed |
| GR00T N1 → N1.7 (NVIDIA, US, 2025–26) | VLM + diffusion transformer, 2–3B | flow matching | open; N1.7 under Apache 2.0 |
| Gemini Robotics 1.0 → 2 (Google DeepMind, UK/US, 2025–26) | Gemini models, cloud and on-device versions | not fully disclosed | closed; reasoning model by API, VLA for partners |
| GO-1 (AgiBot, China, 2025) | VLM + latent action planner | continuous | released with the AgiBot World dataset |
| GR-3 (ByteDance Seed, China, 2025) | VLA co-trained with web data and VR human data | continuous | technical report |
| SmolVLA (Hugging Face, France/US, 2025) | small VLM, 0.45B | flow matching | open |
Three lines of development stand out in 2025–26. Learning from deployment: Physical Intelligence's π*0.6 (November 2025) adds reinforcement learning from the robot's own attempts and from human corrections, and reports roughly halving failure rates on its hardest tasks — making espresso, assembling boxes, folding laundry in real homes. Steerability: π0.7 (April 2026) is conditioned not only on an instruction but on metadata about how to do the task and on subgoal images from a small world model, which lets it train on mixed-quality data, including failures. Whole bodies: Gemini Robotics 2 (July 2026) controls a walking, crouching Apollo humanoid with five-fingered hands, and NVIDIA's GR00T N1.6 and N1.7 target full humanoid bodies. Lumenary Robotics' next lessons take up humanoids and the world models now feeding these policies.
Working with a VLA in practice
Most teams do not pretrain a VLA; they fine-tune an open one on 50–500 demonstrations of their own task, often with low-rank adapters, using openpi, Isaac GR00T or Hugging Face's LeRobot, and serve it from a GPU on or next to the robot. The control loop is short. The sketch below is schematic Python (it needs a real model, cameras and a robot, so it is not run here; each library has its own interface):
# Schematic only: not run here. See openpi, Isaac-GR00T or LeRobot for real interfaces.
while not task_done:
obs = {"images": cameras.read(), "state": robot.joint_positions(), "prompt": "fold the towel"}
chunk = policy.infer(obs) # e.g. 50 future actions, normalised then un-normalised
for action in chunk[:25]: # execute part of the chunk, then re-plan with fresh images
robot.command(action) # a conventional controller tracks each target at high rate
safety_monitor.check(robot) # limits, collisions, emergency stop: see the last lessonExecuting only part of a chunk keeps the policy reactive; Physical Intelligence's "real-time chunking" (2025) goes further, computing the next chunk while the current one runs and constraining it to agree with the actions already committed, so the arm never pauses. Evaluation, safety and deployment covers the monitor in the last line.
What generalises, and how VLAs fail (as of late 2026)
What works. Picking and placing everyday objects never seen in training, in new rooms, from plain instructions, is now routine for the best models. Long, multi-step household tasks in unfamiliar homes work some of the time (π0.5's cleaning; π0.7's appliance tasks). Deformable and dexterous skills — folding laundry, assembling boxes, making coffee — work well after task-specific fine-tuning or reinforcement learning. Transfer between robot bodies is real but partial: π0.7 reports folding laundry zero-shot on a bimanual UR5e setup it had no training data for.
How good, in numbers. Few labs publish success rates against a fixed protocol, and those that do deserve credit. OpenVLA reported beating the 55-billion-parameter RT-2-X by 16.5 percentage points across 29 tasks with seven times fewer parameters. Google DeepMind's Gemini Robotics 2 page reports, for its Apollo humanoid with five-fingered hands, 92% on unscrewing a light bulb but 36% on screwing one in, 44% on tying a bin bag, and 45.7% on picking objects up from the floor. Those are frontier results in July 2026, and they are not the 99.9% a factory needs.
How they fail.
- Ignoring the instruction. Perturbation studies of models that score above 90% on the LIBERO simulation benchmark found that performance can collapse when objects, layouts or camera angles change, and that the models often execute the same motion whatever the instruction says, even when it is replaced by nonsense (LIBERO-PRO and LIBERO-Plus, October 2025). High benchmark scores can reflect memorised trajectories.
- Sensitivity to viewpoint and set-up. A camera moved by a few centimetres, a different table height or new lighting can cut success sharply; LIBERO-Plus measured drops from 95% to below 30%.
- Force and precision. Most VLAs output position targets from images and have no sense of touch, so tight insertions and tasks that depend on contact force remain hard (Dexterous hands and touch).
- Recovery and memory. Policies conditioned on the current frame forget what they have already done; after a slip they may repeat a step or give up. Long tasks compound small error rates.
- Not knowing when they are failing. A VLA's outputs carry no reliable confidence, so a robot can continue confidently in a situation it has never seen. Runtime monitors and safety filters exist for exactly this reason.
- Evaluation itself. Real-robot trials are slow and noisy, and many published comparisons rest on 10–20 trials per task: the statistics of the last lesson of this course matter here.
Common misconceptions
Misconception
"A VLA understands physics because it understands language." The backbone knows what objects are called and roughly where they are. How things move under force comes only from robot data (and, increasingly, from video), and that is the scarcest data in the field.
Misconception
"Bigger backbones make better robots." Size helps semantic generalisation, but OpenVLA (7B) beat RT-2-X (55B), and SmolVLA (0.45B) is competitive on many benchmarks. Data diversity, action representation and control rate matter at least as much, and a bigger model is slower at the one thing a robot cannot wait for.
Misconception
"Tokenised actions are too coarse for precise work." With 256 bins over sensible bounds the quantisation error is tens of micrometres per step, below an arm's repeatability. The real costs of tokens are decoding speed at high control rates and, for naive schemes, weak training signals — which is what FAST and continuous heads address.
Misconception
"The demo video shows what the robot can do." Videos are selected, sometimes sped up, and occasionally teleoperated; responsible labs say which. A success rate over a stated number of trials, under a stated protocol, is the claim to look for.
Misconception
"Flow matching is regression, so it averages modes too." It regresses a velocity at every intermediate noise level, and integrating that field separates the modes. Only a one-step sampler collapses to the mean, as the simulation showed.
Exercises
1. Vision-language-action models
Warm-upHumanA VLA tokenises the wrist's sideways displacement per step with uniform bins over mm, decoding each token to the centre of its bin. For an action inside the bounds, what is the largest possible quantisation error, in micrometres (rounded to the nearest whole number)?
2. Vision-language-action models
Warm-upHumanAn autoregressive VLA on a robot's onboard GPU decodes tokens per second. It uses one token per action dimension per control step (RT-2 and OpenVLA style) and produces one step at a time. What is the highest control rate, in Hz, it can sustain for a 7-dimensional arm?
3. Vision-language-action models
CoreHumanA robot learns to set a cup down on a shelf from 400 demonstrations. Half the operators placed it in the left-hand gap between two jars and half in the right-hand gap; the middle is occupied by a jar. Which output head is most likely to knock the jar over?
4. Vision-language-action models
CoreHumanIn the lesson's simulation, a flow-matching head sampled with a single Euler step put every sample exactly at the mean of the demonstrations. Why?
5. Vision-language-action models
CoreHumanWhich of these are documented weaknesses of vision-language-action models as of 2026? Select all that apply.
6. Vision-language-action models
CoreComputationalImplement OpenVLA-style action binning for a whole dataset. fit_bounds(actions) returns per-dimension bounds at the 1st and 99th percentiles of each column of an (n, d) array. tokenize maps actions to integer tokens in 0 .. n_bins-1 (clipping values outside the bounds, and sending a value equal to high to the last bin). detokenize maps tokens back to the centres of their bins.
7. Vision-language-action models
CoreComputationalA flow-matching action head is sampled by integrating from to . Write integrate(velocity, x0, steps), which takes steps equal Euler steps, evaluating velocity(x, t) at , and returns at . (It must never evaluate the field at , where the ideal field divides by .)
Further reading and resources
- RT-1: Robotics Transformer for Real-World Control at Scale · Brohan et al. (Google) · 2022
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control · Brohan et al. (Google DeepMind) · 2023The paper that named the model class.
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models · Open X-Embodiment Collaboration · 2023
- OpenVLA: An Open-Source Vision-Language-Action Model · Kim, Pertsch et al. · 2024
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success · Kim, Finn, Liang · 2025OpenVLA-OFT, with parallel decoding, action chunks and continuous actions.
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion · Chi et al. · 2023
- Flow Matching for Generative Modeling · Lipman et al. · 2022
- π0: A Vision-Language-Action Flow Model for General Robot Control · Black et al. (Physical Intelligence) · 2024
- FAST: Efficient Action Tokenization for Vision-Language-Action Models · Pertsch et al. (Physical Intelligence) · 2025
- π0.5: a Vision-Language-Action Model with Open-World Generalization · Physical Intelligence · 2025
- Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better · Driess et al. (Physical Intelligence) · 2025
- π*0.6: a VLA That Learns From Experience · Physical Intelligence · 2025
- π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities · Physical Intelligence · 2026
- openpi · Physical IntelligenceOpen weights and code for π0, π0-FAST and π0.5.
- Gemini Robotics: Bringing AI into the Physical World · Gemini Robotics Team (Google DeepMind) · 2025
- Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer · Gemini Robotics Team (Google DeepMind) · 2025
- Gemini Robotics 2 brings whole-body intelligence to robots · Google DeepMind · 2026
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots · NVIDIA · 2025
- Isaac GR00T · NVIDIACode and weights of the GR00T N1 family.
- Helix: A Vision-Language-Action Model for Generalist Humanoid Control · Figure · 2025
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation · Liu et al. (Tsinghua University) · 2024
- AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems · AgiBot World Team · 2025
- GR-3 Technical Report · ByteDance Seed · 2025
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics · Shukor et al. (Hugging Face) · 2025
- LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization · 2025
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models · 2025
- LeRobot · Hugging FaceOpen-source datasets, policies and training code for affordable robots.