Skip to content
LumenaryAcademy
Perception 80 min

Cameras and computer vision for robots

From the pinhole model to pixels, from markers to poses, and from detected features to the first step of visual odometry, with a look at what deep learning now does for robot vision.

Code tested · 5AI-draftedAwaiting review by a roboticist

Your guides: Chloe · The Builder, Margaret · The Experimenter, Gideon · The Safety Engineer

MotivationH

The cheapest, richest and most ambiguous sensor

A tea-picking robot in Assam must find the two leaves and a bud at the tip of each shoot. A warehouse robot in Rotterdam reads the labels on totes. A hospital robot in São Paulo finds its charging dock by a printed square on the wall. All three use cameras, because a camera costing a few dollars delivers millions of measurements thirty or more times a second: colour, texture, text, and the shapes of things. No other sensor comes close in information per unit of cost.

The catch is that a camera throws away one dimension. Every point along a ray from the lens lands on the same pixel, so a photograph cannot tell a small box nearby from a large box far away. Robot vision is largely the art of getting the third dimension back: from known sizes, from markers, from two views, from motion, from learned priors. This lesson builds the geometry, using the frames and transforms of Frames and rigid transforms, and then surveys how features, markers and neural networks turn pixels into the measurements that the filters of this level consume.

IntuitionH

The pinhole camera

Make a tiny hole in one wall of a dark box, and an upside-down image of the scene appears on the opposite wall: every ray of light passes through the one point of the hole. This camera obscura was described by the scholar Ibn al-Haytham in Cairo around 1020, and a modern camera with a lens is, to a very good approximation, the same thing. The hole is the camera centre; the lens only gathers more light.

Put the camera centre at the origin, with the optical axis zz pointing out of the lens. A point at height YY and depth ZZ appears on an image plane at distance ff (the focal length) at height y=fY/Zy = fY/Z, by similar triangles. Nearer things look bigger in proportion to 1/Z1/Z: that is perspective.

OzPYpfZimage planefY/Z
The pinhole model, seen from the side. A point P at height Y and depth Z projects through the camera centre O onto the image plane at height fY/Z. Every point on the dashed ray, near or far, projects to the same image point: depth is lost. (The image plane is drawn in front of the centre, which gives the same geometry without turning the image upside down.)
DefinitionsH

Intrinsics, extrinsics and distortion

Cameras follow the image convention from Frames and rigid transforms: the optical frame CC has zz out of the lens, xx to the right of the image and yy down. Pixel coordinates (u,v)(u, v) start at the top-left corner, uu to the right and vv down.

Definition— Camera intrinsics

The intrinsic matrix

K=(fxscx0fycy001)K = \begin{pmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}

holds the focal lengths in pixels (fx=f/pixel widthf_x = f/\text{pixel width}, fyf_y likewise), the principal point (cx,cy)(c_x, c_y) where the optical axis meets the image, and a skew ss that is zero for every modern sensor. A point Cp=(X,Y,Z){}^{C}\mathbf{p} = (X, Y, Z) projects to λ (u,v,1)T=K Cp\lambda\,(u, v, 1)^{\mathsf T} = K\,{}^{C}\mathbf{p}, with λ=Z\lambda = Z.

Definition— Camera extrinsics and the projection equation

The extrinsics are the camera's pose, for instance WTC{}^{W}T_{C} in the world or BTC{}^{B}T_{C} on the robot body. A world point projects as

λ(uv1)=K [ I    0 ]  CTW(Wp1),CTW=(WTC)−1.\lambda \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = K\,\big[\,I \;\; \mathbf{0}\,\big]\;{}^{C}T_{W} \begin{pmatrix} {}^{W}\mathbf{p} \\ 1 \end{pmatrix}, \qquad {}^{C}T_{W} = \big({}^{W}T_{C}\big)^{-1}.

The 3×43\times4 matrix K [I  0] CTWK\,[I\;\mathbf{0}]\,{}^{C}T_{W} is the camera matrix.

Real lenses bend straight lines, most strongly near the edges of the image. The common Brown–Conrady model distorts the normalised coordinates x=X/Zx = X/Z, y=Y/Zy = Y/Z before KK is applied: with r2=x2+y2r^2 = x^2 + y^2, the radial part is (x,y)↦(1+k1r2+k2r4+k3r6) (x,y)(x, y) \mapsto (1 + k_1 r^2 + k_2 r^4 + k_3 r^6)\,(x, y), plus small tangential terms p1,p2p_1, p_2 for a lens not quite parallel to the sensor. Negative k1k_1 gives barrel distortion (wide-angle lenses), positive gives pincushion. Fisheye lenses, with fields of view of 180° or more, need different models altogether, such as the equidistant or Kannala–Brandt models.

The program projects the corners of a box on the floor through a camera mounted on a robot, exactly as in the frames lesson (2020 cm ahead of the body origin and 4545 cm up, looking forward), then distorts and undistorts a few pixels.

python
import numpy as np

# Intrinsics of a 640 x 480 camera: focal lengths and principal point, in pixels.
fx = fy = 600.0
cx, cy = 320.0, 240.0
K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1.0]])

# Extrinsics: the camera's pose on the robot body, B_T_C (optical frame: z forward, x right, y down).
B_T_C = np.eye(4)
B_T_C[:3, :3] = [[0, 0, 1], [-1, 0, 0], [0, -1, 0]]
B_T_C[:3, 3] = [0.20, 0.0, 0.45]                   # 20 cm ahead of the body origin, 45 cm up
C_T_B = np.linalg.inv(B_T_C)

def project(p_B):
    """Body-frame points (n, 3) -> pixels (n, 2) and depths, for an ideal pinhole camera."""
    p_C = (C_T_B @ np.c_[p_B, np.ones(len(p_B))].T)[:3]
    uvw = K @ p_C
    return (uvw[:2] / uvw[2]).T, p_C[2]

# A 30 cm box on the floor, 2 m ahead of the robot and 0.4 m to its left.
box = np.array([[x, y, z] for x in (2.0, 2.3) for y in (0.25, 0.55) for z in (0.0, 0.3)])
uv, depth = project(box)
print("corner (body frame, m)      pixel (u, v)      depth (m)")
for p, q, d in zip(box, uv, depth):
    print(f"({p[0]:.2f}, {p[1]:.2f}, {p[2]:.2f})   ->   ({q[0]:7.1f}, {q[1]:6.1f})     {d:.2f}")

# Depth is lost: every point on the same ray lands on the same pixel.
lens = B_T_C[:3, 3]                                 # the camera centre, in the body frame
ray = lens + np.outer([1.0, 2.0, 4.0], [1.0, 0.3, -0.1])  # 1, 2 and 4 steps along one direction
print("three points on one ray ->", np.round(project(ray)[0], 2).tolist(), " depths", project(ray)[1].round(2).tolist())
print(f"horizontal field of view: {np.degrees(2 * np.arctan(cx / fx)):.1f} deg")

# Lens distortion (Brown-Conrady radial terms) on normalised coordinates x = X/Z, y = Y/Z.
k1, k2 = -0.25, 0.08                                # a typical wide-ish lens: barrel distortion
def distort(xn):
    r2 = np.sum(xn**2, axis=1, keepdims=True)
    return xn * (1 + k1 * r2 + k2 * r2**2)

def undistort(xd, iters=10):                        # no closed form: fixed-point iteration
    xn = xd.copy()
    for _ in range(iters):
        r2 = np.sum(xn**2, axis=1, keepdims=True)
        xn = xd / (1 + k1 * r2 + k2 * r2**2)
    return xn

pixels = np.array([[320.0, 240.0], [480.0, 240.0], [640.0, 240.0], [640.0, 480.0]])
xn = (pixels - [cx, cy]) / [fx, fy]
moved = distort(xn) * [fx, fy] + [cx, cy]
back = undistort(distort(xn)) * [fx, fy] + [cx, cy]
print("\nideal pixel      distorted pixel     shift (px)   recovered by undistortion")
for a, b, c in zip(pixels, moved, back):
    print(f"({a[0]:5.0f}, {a[1]:4.0f})   ({b[0]:6.1f}, {b[1]:6.1f})   {np.hypot(*(b - a)):8.1f}       ({c[0]:.2f}, {c[1]:.2f})")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The box's near face, 1.8 m from the lens, spans 100 pixels across; the far face, 2.1 m away, spans 86. Three points at depths 1, 2 and 4 m along one ray all land on pixel (140,300)(140, 300). With fx=600f_x = 600 px across a 640-pixel image, the horizontal field of view is 2arctan⁡(320/600)=56.1°2\arctan(320/600) = 56.1°. The distortion barely moves a pixel near the centre (2.8 px halfway to the edge) but moves the image corner by 38 px. Undistortion has no closed form; the fixed-point iteration recovers the ideal pixels to within a hundredth of a pixel.

ideal pinholebarrel distortion
Barrel distortion with the lens of the program (k₁ = −0.25, k₂ = 0.08), applied to a square grid that fills the 640 × 480 image. Straight lines through the centre stay straight; all others bow outwards, and the corners are pulled in by about 38 pixels.
ExplanationH

Calibration: measuring K and the distortion

Intrinsics are measured, not looked up: the nominal focal length on a datasheet is good to a few per cent, and the principal point is rarely exactly at the image centre. Zhang's method (2000), used by OpenCV, MATLAB and most other tools, needs only a flat pattern of known geometry, typically a printed checkerboard or a ChArUco board (a checkerboard with markers, so partial views still work):

  1. Photograph the board in 10 to 30 different orientations, filling different parts of the image, especially the corners where distortion is strongest.
  2. In each image, detect the board's corners with sub-pixel accuracy. Because the board is flat, each view is related to the board by a 3×33\times3 homography H=K [r1  r2  t]H = K\,[\mathbf{r}_1\;\mathbf{r}_2\;\mathbf{t}], estimated from the corner correspondences.
  3. The columns r1,r2\mathbf{r}_1, \mathbf{r}_2 of a rotation are orthonormal, which gives two linear equations per view in the entries of K−TK−1K^{-\mathsf T}K^{-1}. With three or more views, solve for KK in closed form; then recover each view's pose.
  4. Refine everything (intrinsics, distortion and every pose) together by nonlinear least squares, minimising the reprojection error: the distance in pixels between each detected corner and where the model projects it.

A good calibration has an RMS reprojection error of a fraction of a pixel; one well above a pixel usually means a blurred image, a board that was not flat, or too few views near the edges. In OpenCV the whole procedure is a few calls (shown for reading, not run here; it needs OpenCV and real images):

python
# OpenCV calibration sketch: not run here (needs OpenCV and a folder of board images).
import glob
import cv2
import numpy as np

cols, rows, square = 9, 6, 0.025                       # inner corners, and square size in metres
board = np.zeros((cols * rows, 3), np.float32)
board[:, :2] = np.mgrid[0:cols, 0:rows].T.reshape(-1, 2) * square

object_points, image_points = [], []
for name in sorted(glob.glob("calib/*.png")):
    grey = cv2.imread(name, cv2.IMREAD_GRAYSCALE)
    found, corners = cv2.findChessboardCorners(grey, (cols, rows))
    if found:
        corners = cv2.cornerSubPix(grey, corners, (11, 11), (-1, -1),
                                   (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3))
        object_points.append(board)
        image_points.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(object_points, image_points, grey.shape[::-1], None, None)
print("RMS reprojection error (px):", rms)
print("K =\n", K, "\ndistortion (k1, k2, p1, p2, k3) =", dist.ravel())

Robots need more than intrinsics. The extrinsic calibration BTC{}^{B}T_{C} between a camera and the robot body, or between a camera and a lidar, is estimated from shared views of targets; a camera on an arm's wrist needs hand–eye calibration; and a camera fused with an IMU needs their relative pose and the time offset between their clocks, which open tools such as ETH Zurich's Kalibr estimate together. As the frames lesson showed, a 1° error in a mounting rotation misplaces a point 10 m away by 17 cm, so this work is never wasted.

ExplanationH

Features and matching

To recognise a place, follow motion or build a map, a robot must find the same physical point in different images. The classical answer is local features.

  • A detector finds points that can be located precisely: corners (Harris, 1988; FAST, 2006) or blob centres at a range of scales (SIFT, 1999 and 2004). A flat wall or a uniform sky has none; a bookshelf has thousands.
  • A descriptor summarises the patch around each point as a vector, designed to change little with viewpoint, lighting and rotation. SIFT's is 128 numbers built from gradient directions. ORB (2011), designed for speed and free of patents, uses 256 bits, each the result of comparing two pixel intensities, so two descriptors are compared by counting differing bits (the Hamming distance), which processors do in one instruction.
  • Matching pairs each descriptor with its nearest neighbour in the other image. Many nearest neighbours are wrong, especially on repetitive textures such as tiles, windows and shelving. David Lowe's ratio test keeps a match only if the best candidate is clearly better than the second best (say, distance less than 0.8 times the second distance); an ambiguous match is worse than none.
  • Geometric verification fits a model (a homography, an essential matrix, a pose) to the surviving matches with RANSAC (Fischler and Bolles, 1981): repeatedly fit the model to a random minimal sample, count how many matches agree, keep the best. Outliers that survived the ratio test are rejected because they disagree with the geometry.

Since about 2018, learned features have joined the classical ones. SuperPoint (2018) learns a detector and descriptor together; SuperGlue (2020) and its faster successor LightGlue (2023) match features with attention, the mechanism of Lumenary CS's attention and transformers lesson, reasoning about all the points at once; LoFTR (2021) matches densely without a detector. They are markedly more robust to large changes of viewpoint, season and lighting, and are standard in visual localisation research; ORB remains common on small robots because it is cheap and predictable.

ComputationC

Fiducial markers and pose from known points

When a robot needs a pose it can trust (a dock, a pallet, a calibration target, a test rig), it is easiest to put a known pattern in the world. Fiducial markers such as AprilTag (Edwin Olson, University of Michigan, 2011; later versions 2016 and 2019) and ArUco (University of Córdoba) are black-and-white squares whose interior encodes an identity with error-correcting codes. A detector finds quadrilaterals, reads the code and reports the identity and the four corner pixels to sub-pixel accuracy.

Four corners of a square of known size are enough to recover the marker's full pose CTM{}^{C}T_{M}: this is the Perspective-n-Point (PnP) problem. Closed-form solvers exist (P3P and its relatives, and IPPE for planar targets), but the most instructive method is the one every solver finishes with: least squares on the reprojection error. Start from a rough guess; repeatedly linearise the projection of the four corners around the current pose (a 8×68\times6 Jacobian: 8 pixel coordinates, 6 pose parameters), solve for the small motion that best reduces the error, and apply it. This is Gauss–Newton, the same method as numerical inverse kinematics, with the pose update applied as a small rotation vector through Rodrigues' formula, as in Rotations and quaternions.

python
import numpy as np

K = np.array([[600.0, 0, 320], [0, 600.0, 240], [0, 0, 1]])
s = 0.16 / 2                                           # a 16 cm square marker; corners in its own frame M
corners_M = np.array([[-s, s, 0], [s, s, 0], [s, -s, 0], [-s, -s, 0]])

def rodrigues(w):
    """Rotation matrix exp([w]x) for a rotation vector w (axis times angle)."""
    th = np.linalg.norm(w)
    if th < 1e-12:
        return np.eye(3)
    k = w / th
    Kx = np.array([[0, -k[2], k[1]], [k[2], 0, -k[0]], [-k[1], k[0], 0]])
    return np.eye(3) + np.sin(th) * Kx + (1 - np.cos(th)) * Kx @ Kx

def project(R, t, P):                                   # pose C_T_M = (R, t): marker in the camera frame
    pc = P @ R.T + t
    uv = pc @ K.T
    return uv[:, :2] / uv[:, 2:]

def solve_pnp(uv, R, t, iters=10, verbose=False):
    """Gauss-Newton on the reprojection error, updating the pose by small rigid motions."""
    for it in range(iters):
        r = (uv - project(R, t, corners_M)).ravel()      # 8 residuals, in pixels
        if verbose:
            print(f"  iteration {it}: RMS reprojection error {np.sqrt(np.mean(r**2)):9.3f} px")
        J = np.zeros((8, 6))
        for j in range(6):                               # numerical Jacobian, one column per pose parameter
            d = np.zeros(6); d[j] = 1e-6
            Rj, tj = rodrigues(d[:3]) @ R, rodrigues(d[:3]) @ t + d[3:]
            J[:, j] = (project(Rj, tj, corners_M) - project(R, t, corners_M)).ravel() / 1e-6
        delta = np.linalg.lstsq(J, r, rcond=None)[0]     # solve J delta = r in the least-squares sense
        dR = rodrigues(delta[:3])
        R, t = dR @ R, dR @ t + delta[3:]                # apply the small motion on the left
    return R, t

# The true pose: the marker 1.5 m away, turned 30 deg about the vertical and tilted back 20 deg.
R_true = rodrigues(np.radians([0, 30, 0])) @ rodrigues(np.radians([-20, 0, 0]))
t_true = np.array([0.10, -0.05, 1.5])
rng = np.random.default_rng(0)
uv = project(R_true, t_true, corners_M) + rng.normal(0, 0.5, (4, 2))     # detected corners, 0.5 px noise
print("solving from a guess of 'facing the camera, 1 m away':")
R, t = solve_pnp(uv, np.eye(3), np.array([0, 0, 1.0]), iters=7, verbose=True)
ang = np.degrees(np.arccos(np.clip((np.trace(R.T @ R_true) - 1) / 2, -1, 1)))
print(f"position error {1000 * np.linalg.norm(t - t_true):.1f} mm, orientation error {ang:.2f} deg")

# How does the answer degrade with distance? 200 noisy detections at each range.
print("\ndistance   sd of estimated distance   RMS orientation error")
for dist in (0.5, 1.5, 3.0, 6.0):
    tt = np.array([0.0, 0.0, dist])
    zs, angs = [], []
    for _ in range(200):
        uv = project(R_true, tt, corners_M) + rng.normal(0, 0.5, (4, 2))
        R, t = solve_pnp(uv, R_true, tt)
        zs.append(t[2])
        angs.append(np.degrees(np.arccos(np.clip((np.trace(R.T @ R_true) - 1) / 2, -1, 1))))
    print(f"  {dist:3.1f} m          {1000 * np.std(zs):7.1f} mm              {np.sqrt(np.mean(np.square(angs))):5.2f} deg")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

From a poor first guess (facing the camera at 1 m, when the marker is at 1.5 m and turned 30°), the error falls from 37 px to 0.22 px in four iterations and then stops improving: 0.22 px is the level of the pixel noise, so the remaining error is the measurement's, not the solver's. The pose is within 6 mm and 0.34°.

The second table is the important one for system design. With the same 0.5-pixel noise, the standard deviation of the estimated distance grows from 1.3 mm at 0.5 m to 205 mm at 6 m, roughly with the square of the distance, and the orientation error grows in proportion to the distance. At 6 m this 16 cm marker spans only 16 pixels, and the four corners barely constrain its tilt.

ExplanationH

Deep learning for detection and segmentation

Markers and features find points. Robots also need to know what things are: a person, a pallet, a ripe tomato, a doorway, the free floor. Since around 2012 this has been the domain of deep neural networks, trained on large labelled datasets; Lumenary CS's neural networks, deep architectures and attention and transformers lessons explain how they work. As of 2026, the main tools a robotics engineer reaches for are:

  • Object detection: a box and a class label for each object. Convolutional detectors such as the YOLO family (from 2015, developed since by several groups) run at video rate on small embedded GPUs; transformer detectors (DETR, 2020, and its successors) remove hand-tuned post-processing.
  • Segmentation: a class or instance label for every pixel (Mask R-CNN, 2017), which a robot needs to grasp an object or to know which pixels are drivable floor. Meta's Segment Anything (2023) and SAM 2 (2024) segment almost any object from a click or a box prompt, without training on that object.
  • Open-vocabulary detection: models trained on images paired with text, such as OWL-ViT and Grounding DINO, find objects described in words ("the blue mug next to the kettle") rather than from a fixed list of classes. They connect robot perception to the language interfaces of the vision-language-action models of the Frontier level.
  • General-purpose visual features learned without labels, such as DINOv2 (2023), are used as a starting point for many downstream tasks, from place recognition to grasp prediction.
  • Monocular depth: networks that estimate depth from a single image, covered in Depth sensing and point clouds.

Engineering with these models is different from engineering with geometry. Their outputs carry no calibrated uncertainty unless you add it; they fail on images unlike their training data (night, rain, a new warehouse's lighting, an unusual object), and they can fail confidently. Robots therefore combine them with geometry: a detector proposes "pallet here", depth or a model fit gives its pose, and a tracker (a Kalman filter per object) smooths it over time and rejects one-frame mistakes. Latency matters too: a network taking 100 ms on an embedded computer means the robot is reacting to where things were.

ComputationC

Two views: the first step of visual odometry

Visual odometry (VO) estimates a camera's motion from the images it takes: match features between consecutive frames, find the relative pose that explains the matches, and chain the relative poses together. For a calibrated camera, matches between two views obey the epipolar constraint

x2T E x1=0,E=[t]×R,\mathbf{x}_2^{\mathsf T}\,E\,\mathbf{x}_1 = 0, \qquad E = [\mathbf{t}]_\times R,

where x1,x2\mathbf{x}_1, \mathbf{x}_2 are a point's normalised coordinates K−1(u,v,1)TK^{-1}(u, v, 1)^{\mathsf T} in the two images, (R,t)(R, \mathbf{t}) is the motion, and [t]×[\mathbf{t}]_\times is the cross-product matrix. EE is the essential matrix. Each match gives one equation that is linear in the nine entries of EE, so eight matches determine it (the eight-point algorithm; Nistér's five-point algorithm, used inside RANSAC in practice, needs only five). EE factors into four possible motions, and the right one is the one that puts the points in front of both cameras.

python
import numpy as np

rng = np.random.default_rng(1)
K = np.array([[600.0, 0, 320], [0, 600.0, 240], [0, 0, 1]])

def rot_y(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

def skew(v):
    return np.array([[0, -v[2], v[1]], [v[2], 0, -v[0]], [-v[1], v[0], 0]])

# 3-D points in front of camera 1, and camera 2's motion: 0.5 m to the right and 5 deg turn.
P1 = np.column_stack([rng.uniform(-2, 2, 40), rng.uniform(-1, 1, 40), rng.uniform(4, 10, 40)])
R, t = rot_y(np.radians(5)), np.array([-0.5, 0.0, 0.05])     # x2 = R x1 + t (camera 2 coordinates)

def pixels(P):
    uv = P @ K.T
    return uv[:, :2] / uv[:, 2:]

uv1, uv2 = pixels(P1), pixels(P1 @ R.T + t)
uv1n, uv2n = uv1 + rng.normal(0, 0.3, uv1.shape), uv2 + rng.normal(0, 0.3, uv2.shape)

# Scale ambiguity: double the scene AND the motion, and the images do not change at all.
change = max(np.abs(pixels(2 * P1) - uv1).max(), np.abs(pixels(2 * P1 @ R.T + 2 * t) - uv2).max())
print(f"largest pixel change when the scene and the motion are both doubled: {change:.6f} px")

# Eight-point algorithm on normalised coordinates: each match gives one linear equation x2^T E x1 = 0.
x1 = np.c_[uv1n, np.ones(40)] @ np.linalg.inv(K).T
x2 = np.c_[uv2n, np.ones(40)] @ np.linalg.inv(K).T
A = np.einsum("ni,nj->nij", x2, x1).reshape(40, 9)
E = np.linalg.svd(A)[2][-1].reshape(3, 3)              # the null vector of A
U, S, Vt = np.linalg.svd(E)
E = U @ np.diag([1, 1, 0]) @ Vt                        # an essential matrix has singular values (s, s, 0)

# Four (R, t) pairs fit E; keep the one that puts the points in front of both cameras.
W = np.array([[0, -1, 0], [1, 0, 0], [0, 0, 1]])
best = None
for Rc in (U @ W @ Vt, U @ W.T @ Vt):
    Rc = Rc * np.sign(np.linalg.det(Rc))
    for tc in (U[:, 2], -U[:, 2]):
        # triangulate each point by least squares: depth1 * R x1 + t = depth2 * x2
        good = 0
        for a, b in zip(x1, x2):
            M = np.column_stack([Rc @ a, -b])
            d1, d2 = np.linalg.lstsq(M, -tc, rcond=None)[0]
            good += (d1 > 0) and (d2 > 0)
        if best is None or good > best[0]:
            best = (good, Rc, tc)
good, R_est, t_est = best
rot_err = np.degrees(np.arccos(np.clip((np.trace(R_est.T @ R) - 1) / 2, -1, 1)))
dir_err = np.degrees(np.arccos(t_est @ t / np.linalg.norm(t)))
print(f"points in front of both cameras: {good} of 40")
print(f"rotation error {rot_err:.3f} deg; translation direction error {dir_err:.2f} deg")
print(f"estimated translation {np.round(t_est, 3)} has length {np.linalg.norm(t_est):.1f}: the true length,"
      f" {np.linalg.norm(t):.3f} m, cannot be recovered from the images")
print("epipolar residuals |x2^T E x1| (median):", f"{np.median(np.abs(np.einsum('ni,ij,nj->n', x2, E, x1))):.1e}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Doubling the scene and the camera's motion together changes no pixel at all, so no algorithm working on these images alone can measure the length of t\mathbf{t}. The eight-point algorithm recovers the rotation to 0.3° and the direction of travel to 1.5° from 40 noisy matches, and returns a translation of length 1 by convention. This scale ambiguity is fundamental to a single camera: a monocular VO system knows the shape of its path but not its size, and the unknown scale slowly drifts as errors accumulate. It is resolved by a second camera with a known baseline (stereo), by an IMU, whose accelerometer measures metres per second squared, by wheel odometry, or by objects of known size. Visual-inertial odometry, a camera fused with an IMU, is what phones, headsets and most drones use; the filters and optimisers that do it appear in Kalman filters and SLAM.

A practical VO front end, sketched with OpenCV (for reading, not run here):

python
# Monocular VO step with OpenCV: not run here (needs OpenCV and two images).
import cv2
import numpy as np

orb = cv2.ORB_create(nfeatures=2000)
kp1, d1 = orb.detectAndCompute(img1, None)
kp2, d2 = orb.detectAndCompute(img2, None)
pairs = cv2.BFMatcher(cv2.NORM_HAMMING).knnMatch(d1, d2, k=2)
good = [m for m, n in pairs if m.distance < 0.8 * n.distance]           # Lowe's ratio test
p1 = np.float32([kp1[m.queryIdx].pt for m in good])
p2 = np.float32([kp2[m.trainIdx].pt for m in good])
E, inliers = cv2.findEssentialMat(p1, p2, K, method=cv2.RANSAC, prob=0.999, threshold=1.0)
n, R, t, mask = cv2.recoverPose(E, p1, p2, K, mask=inliers)              # t has unit length
ExplanationH

In practice

Several details separate a camera that works on the bench from one that works on a robot.

  • Shutter. Most inexpensive sensors have a rolling shutter: rows are exposed one after another over several milliseconds, so a fast-turning robot sees vertical lines leaning. Global-shutter sensors expose all pixels at once and are preferred for VO and for anything fast.
  • Exposure and motion blur. In dim light the exposure lengthens, and a moving robot smears features; auto-exposure that changes between frames also confuses matching. Robots often fix exposure limits and add lighting.
  • Time. A camera image must be stamped with when it was exposed, not when it arrived, and synchronised with the IMU and odometry; a 20 ms offset at 1 m/s is a 2 cm error before any vision is done.
  • Software. In ROS 2, a camera driver publishes sensor_msgs/Image with a CameraInfo message holding KK and the distortion, and the image_pipeline packages rectify images; OpenCV provides most classical algorithms, and marker detection is available in both OpenCV and the AprilTag library.
  • Safety and privacy. Cameras on robots in homes, hospitals and streets record people. Good practice processes images on the robot, stores as little as possible, and follows local data-protection law.
Common misconceptionsH

Common misconceptions

Misconception

"The focal length is 4 mm, so fx=4f_x = 4." KK uses focal lengths in pixels: the focal length divided by the pixel pitch. A 4 mm lens on a sensor with 3 µm pixels has fx≈1333f_x \approx 1333 px. Calibration measures this number directly.

Misconception

"The image yy axis points up." In image and optical-frame conventions it points down, and vv grows downwards from the top-left corner. Mixing this up flips poses upside down, and it is one of the most common bugs when connecting vision code to robot frames.

Misconception

"A single camera can measure distance if the software is clever enough." Not from geometry alone: the program showed that scaling the world and the motion together leaves every pixel unchanged. Monocular depth networks estimate distance from learned priors (how big cars and doors usually are), which works often and fails without warning on unusual scenes.

Misconception

"Distortion only matters for fisheye lenses." Even ordinary lenses move pixels near the image corners by tens of pixels, as the first program showed. Ignoring distortion bends straight walls in a map and biases every pose estimated from points near the edges.

Exercises

Exercises

1. Cameras and computer vision for robots

Warm-upHuman

A camera has fx=fy=500f_x = f_y = 500 px and principal point (cx,cy)=(320,240)(c_x, c_y) = (320, 240), with no distortion. A point has camera-frame coordinates Cp=(0.4,−0.2,2.0){}^{C}\mathbf{p} = (0.4, -0.2, 2.0) m (optical frame: xx right, yy down, zz forward). What is its pixel coordinate uu?

2. Cameras and computer vision for robots

CoreHuman

A camera's images are 1280 pixels wide, its principal point is at the centre, and its horizontal field of view is 90°90°. What is fxf_x in pixels?

3. Cameras and computer vision for robots

Warm-upHuman

Which of these are intrinsic parameters of a camera, that is, they do not change when the robot carrying the camera drives around or when the camera is unbolted and mounted elsewhere? Select all that apply.

Choose all that apply

4. Cameras and computer vision for robots

CoreHuman

A robot with the lesson's camera (fx=600f_x = 600 px) must read a 16 cm AprilTag on a charging dock. Roughly how many pixels wide does the tag appear when it faces the camera 4 m away?

5. Cameras and computer vision for robots

CoreHuman

A drone runs monocular visual odometry and its trajectory has the right shape but the wrong size. Which of these can supply the missing metric scale? Select all that apply.

Choose all that apply

6. Cameras and computer vision for robots

CoreComputational

Write project_points(K, C_T_W, P_W). K is the 3×33\times3 intrinsic matrix, C_T_W the 4×44\times4 transform that maps world coordinates into the camera's optical frame (the pose of the world in the camera frame), and P_W an (n,3)(n, 3) array of world points. Return an (n,2)(n, 2) array of pixels (u,v)(u, v), with nan in the rows of points that are behind the camera (depth ≤0\le 0).

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Cameras and computer vision for robots

StretchComputational

Implement Lowe's ratio test for binary descriptors like ORB's. desc_a and desc_b are uint8 arrays of shape (n,32)(n, 32): 256 bits per descriptor. For each row ii of desc_a, find the nearest and second-nearest rows of desc_b by Hamming distance (the number of differing bits) and keep the pair (i,jbest)(i, j_{\text{best}}) only if the best distance is less than ratio times the second best. Return the list of kept pairs, in increasing order of ii.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources