Why is picking things up so hard?
A three-year-old can pick a toy out of a toy box without looking very carefully. A warehouse robot facing a tote of 40 different items, a shampoo bottle on its side, a bag of socks, a boxed phone charger and a book wedged against the wall, has to solve a chain of problems in about a second: what is in the bin, where exactly each item is, how to hold the one it wants so that it will not slip or swing, which path brings the gripper there without hitting the bin or the other items, and finally whether it actually has the item. Each step fails a few per cent of the time, and the failures multiply.
From 2015 to 2017 Amazon ran an open competition, the Amazon Picking Challenge (later the Amazon Robotics Challenge), to push exactly this problem; the winning teams came from TU Berlin (2015), TU Delft (2016) and the Australian Centre for Robotic Vision (2017), and the leading entries relied heavily on suction. Since then, picking robots have moved from contests into warehouses in many countries, and grasping has become one of the main proving grounds for robot learning.
This lesson builds the core ideas from the contact upwards: grippers, the friction cone, force closure with an implemented test and a quality score, the pick-and-place pipeline with a simulation of why verification matters, and the learned grasp predictors used in 2026.
Grippers
The end effector shapes everything downstream. Four families cover most robots.
- Parallel-jaw grippers: two fingers that close along a slide, driven electrically or pneumatically. Simple, strong, precise and easy to model (a grasp is two contacts and a width), they are the default in research and on machine-tending arms. Makers in many countries build them, for example Schunk in Germany, Robotiq in Canada and SMC in Japan.
- Suction grippers: one or more vacuum cups, evacuated by a pump or by an ejector that uses compressed air. They need only one accessible flat-ish face, which makes them the workhorse of logistics, where most items are boxes, bottles and sealed bags. They fail on porous, very curved, dusty or tiny items.
- Soft grippers: fingers made of rubber-like materials, often inflated with air, or fin-shaped fingers that wrap around an object when pressed, or a bag of granules that flows around an object and then stiffens when the air is pumped out ("jamming"). They are forgiving of uncertainty and gentle with food and fruit, at the cost of precision and strength.
- Multi-fingered hands: three to five articulated fingers, such as the Shadow Dexterous Hand (United Kingdom), the Allegro Hand (South Korea) and the low-cost open-source LEAP Hand from Carnegie Mellon University (2023). They can reorient objects in the hand, not just hold them, and are the subject of Dexterous manipulation.
Contacts, friction cones and wrenches
To reason about a grasp we model each finger's touch as a contact at a point on the object's surface, with inward unit normal . What forces can the finger apply there?
Definition— Contact models and the friction cone
- A frictionless point contact can only push along the normal: with .
- A point contact with friction (a "hard finger") can push with any force inside the friction cone: the normal component and the tangential component , where is the Coulomb friction coefficient. The cone's half-angle is .
- A soft-finger contact adds a limited torque about the normal, from the area of a squashed fingertip.
In the plane, the friction cone is a wedge spanned by two edge forces, and (with the tangent), and every allowed force is a non-negative combination of them. In 3-D, the round cone is often replaced by a pyramid of 4 to 8 edges.
A force applied at acts on the object as a wrench: the force together with its torque about the object's reference point, . In the plane a wrench has three components, , with ; in space it has six. Stacking the contacts' contributions gives the grasp map , from all the contact forces to the net wrench on the object.
Definition— Force closure
A grasp is in force closure if the fingers, by applying suitable forces inside their friction cones, can balance any external wrench on the object: for every there are allowed contact forces with . (When the contacts are frictionless and this still holds, the grasp is in form closure: the geometry alone pins the object.)
Force closure is what lets a grasp hold an object whatever the motion: gravity in any direction, the jolt of a fast move, a collision with the bin wall. The fingers do not need to know the disturbance in advance; they need only be able to resist it.
Theorem— Convex-hull test for force closure
Let be the wrenches of all the friction-cone edges of a grasp, each with unit normal component. The grasp is in force closure if and only if the origin lies in the interior of the convex hull of .
Proof
Every allowed contact force is a non-negative combination of its cone's edges, and the wrench is linear in the forces, so the set of wrenches the grasp can produce is the cone . Force closure says this cone is the whole wrench space.
If the origin is interior to the hull, some ball of radius around it lies in the hull, so every wrench of size at most is a convex (hence non-negative) combination of the ; any other wrench is a positive multiple of one of those, so it is produced too. Conversely, if the origin is not interior, a supporting hyperplane separates it from the hull's interior: there is a direction with for all (because each lies on the hull's side of the plane). Then every producible wrench has , so the wrench cannot be produced, and the grasp is not force closure.
∎
Testing force closure in code
The program builds the edge wrenches of a planar grasp and then measures how far inside the convex hull the origin sits. With a few points we can find the hull's faces by brute force: a triangle of points is a face if every other point lies on one side of its plane. The smallest distance from the origin to a face is the Ferrari–Canny quality (1992): the largest disturbance wrench, in the worst direction, that the grasp can resist with a total normal force of 1. Positive means force closure; larger is more robust. Torques are divided by a characteristic length (here 5 cm) so that forces and torques can be compared.
import itertools
import numpy as np
def contact_wrenches(points, normals, mu, rho):
"""Edges of each planar friction cone, as wrenches (fx, fy, torque / rho).
Each edge force is n + mu t or n - mu t: unit normal component, the most tangential force allowed."""
W = []
for p, n in zip(np.asarray(points, float), np.asarray(normals, float)):
n = n / np.linalg.norm(n)
t = np.array([-n[1], n[0]]) # the tangent direction
for f in (n + mu * t, n - mu * t) if mu > 0 else (n,):
W.append([f[0], f[1], (p[0] * f[1] - p[1] * f[0]) / rho])
return np.array(W)
def grasp_quality(W, tol=1e-12):
"""Distance from the origin to the boundary of conv(W), negative or zero if the origin is not
strictly inside. Positive means force closure; the value is the Ferrari-Canny epsilon."""
if np.linalg.matrix_rank(W - W.mean(axis=0), tol=1e-9) < 3:
return 0.0 # a flat hull has no interior
best = np.inf
for i, j, k in itertools.combinations(range(len(W)), 3):
n = np.cross(W[j] - W[i], W[k] - W[i])
if np.linalg.norm(n) < 1e-12:
continue
n /= np.linalg.norm(n)
s = W @ n - n @ W[i]
if np.all(s <= tol): # every point on one side: a facet
best = min(best, n @ W[i])
elif np.all(s >= -tol):
best = min(best, -(n @ W[i]))
return best
rho = 0.05 # torques are divided by a 5 cm lever
grasps = {
"two fingers, opposite faces, mu = 0.5": ([(-0.05, 0), (0.05, 0)], [(1, 0), (-1, 0)], 0.5),
"the same, frictionless (mu = 0)": ([(-0.05, 0), (0.05, 0)], [(1, 0), (-1, 0)], 0.0),
"two fingers offset 6 cm, mu = 0.5": ([(-0.05, 0.03), (0.05, -0.03)], [(1, 0), (-1, 0)], 0.5),
"two fingers offset 6 cm, mu = 0.8": ([(-0.05, 0.03), (0.05, -0.03)], [(1, 0), (-1, 0)], 0.8),
"three fingers on a disc, mu = 0.2": ([(0.04 * np.cos(a), 0.04 * np.sin(a)) for a in (0, 2.0944, 4.1888)],
[(-np.cos(a), -np.sin(a)) for a in (0, 2.0944, 4.1888)], 0.2),
"three fingers on a disc, mu = 0": ([(0.04 * np.cos(a), 0.04 * np.sin(a)) for a in (0, 2.0944, 4.1888)],
[(-np.cos(a), -np.sin(a)) for a in (0, 2.0944, 4.1888)], 0.0),
"four frictionless fingers on a box": ([(-0.05, 0.03), (0.05, 0.01), (0.0, 0.05), (0.02, -0.05)],
[(1, 0), (-1, 0), (0, -1), (0, 1)], 0.0),
}
for name, (pts, nrm, mu) in grasps.items():
q = grasp_quality(contact_wrenches(pts, nrm, mu, rho))
print(f"{name:40} quality {q:+.4f} -> {'force closure' if q > 1e-9 else 'NOT force closure'}")Each line teaches something:
- Two fingers squeezing opposite faces with are in force closure (). Without friction they are not: they can only push along one line, so they cannot resist a sideways push or a twist.
- Offset the fingers so that the line between them is tilted by from the faces' normals, and it fails at (friction angle 26.6°) but succeeds at (38.7°). Friction must be able to cover the tilt.
- Three fingers at 120° around a disc are force closure with even modest friction, but frictionless they cannot resist twisting, because every normal passes through the centre.
- Four frictionless fingers, placed on a box so that their normals do not all meet, achieve form closure, matching Chloe's count of .
Theorem— Two-finger planar force closure (Nguyen, 1988)
A planar grasp with two point contacts with friction is in force closure if and only if the segment joining the two contacts lies strictly inside both friction cones, that is, it makes an angle smaller than with each inward normal.
The idea of the proof: if the segment lies inside both cones, the fingers can squeeze along it with any internal force, and adding small extra forces within the cones (which have room around the segment) resists any small wrench; if it lies outside a cone, one finds a twist or push that neither finger can oppose. The antipodal grasps that parallel-jaw grippers make are the case where the two normals are opposite and aligned with the segment.
Choosing a grasp
Grasp selection combines quality with what the hardware allows. Below, a parallel-jaw gripper that opens to 9 cm must hold an oval object 12 cm by 6 cm (a bar of soap, say) with . We try pairs of opposite points around the oval, compute the angle between the finger line and the normal (Nguyen's condition), the quality, and the opening width required.
import itertools
import numpy as np
def contact_wrenches(points, normals, mu, rho):
W = []
for p, n in zip(np.asarray(points, float), np.asarray(normals, float)):
n = n / np.linalg.norm(n)
t = np.array([-n[1], n[0]])
for f in (n + mu * t, n - mu * t) if mu > 0 else (n,):
W.append([f[0], f[1], (p[0] * f[1] - p[1] * f[0]) / rho])
return np.array(W)
def grasp_quality(W, tol=1e-12):
if np.linalg.matrix_rank(W - W.mean(axis=0), tol=1e-9) < 3:
return 0.0
best = np.inf
for i, j, k in itertools.combinations(range(len(W)), 3):
n = np.cross(W[j] - W[i], W[k] - W[i])
if np.linalg.norm(n) < 1e-12:
continue
n /= np.linalg.norm(n)
s = W @ n - n @ W[i]
if np.all(s <= tol):
best = min(best, n @ W[i])
elif np.all(s >= -tol):
best = min(best, -(n @ W[i]))
return best
# An oval object (a bar of soap, 12 cm x 6 cm); a parallel-jaw gripper that opens to 9 cm; mu = 0.4.
a, b, mu, rho, max_open = 0.06, 0.03, 0.4, 0.05, 0.09
print(" angle width line-vs-normal friction angle quality verdict")
best = None
for deg in range(0, 91, 15):
t = np.radians(deg)
p1 = np.array([a * np.cos(t), b * np.sin(t)])
p2 = -p1 # the opposite point through the centre
n1 = -np.array([np.cos(t) / a, np.sin(t) / b]) # inward normals
n1 /= np.linalg.norm(n1)
n2 = -n1
width = np.linalg.norm(p1 - p2)
u = (p2 - p1) / width
off = np.degrees(np.arccos(np.clip(u @ n1, -1, 1))) # angle between the finger line and the normal
q = grasp_quality(contact_wrenches([p1, p2], [n1, n2], mu, rho))
ok = q > 1e-9 and width <= max_open
verdict = "ok" if ok else ("too wide" if q > 1e-9 else "slips")
print(f"{deg:5d} {100 * width:4.1f} cm {off:8.1f} deg {np.degrees(np.arctan(mu)):8.1f} deg {q:+.4f} {verdict}")
if ok and (best is None or q > best[1]):
best = (deg, q)
print(f"chosen grasp: contacts at {best[0]} deg, quality {best[1]:.4f}")The two tests agree line by line: wherever the finger line is within the 21.8° friction angle the quality is positive, and wherever it is not (33° and 36.9°) the grasp slips. The best grasp by quality, across the long axis (), needs a 12 cm opening the gripper does not have; the chosen grasp is across the short axis, at 6 cm. The grasps at 15° and 75° are force closure only just (–): legal, but a little error in where the fingers land, or a slightly lower , and they slip. Real grasp planners therefore prefer grasps whose quality is robust to the perception error they expect.
The pick-and-place pipeline
An industrial picking cell runs the same loop thousands of times a day.
- Perceive. Cameras and depth sensors over the bin produce images and point clouds (Depth and point clouds); a segmentation network separates the items; for known items, a pose estimator finds each one's 6-D pose. Shiny, transparent and black items are hard for depth sensors, and clutter hides most surfaces.
- Select a grasp. Candidate grasps are generated (suction points on flat patches, antipodal pairs for jaws), filtered by reachability and collisions, and ranked by an analytic quality like the one above or by a learned predictor.
- Plan the motion. A collision-free path to a pre-grasp pose above the item, then a straight approach, is found with inverse kinematics and a planner (Inverse kinematics, Sampling-based planning); open-source systems often use MoveIt.
- Execute. The robot approaches, makes contact under force or impedance control (Force and impedance control) so that a few millimetres of perception error do not become a crash, grips, lifts and transports.
- Verify. Did it work? Suction systems read the vacuum pressure; jaw grippers check the finger width and motor current; a wrist force sensor or a scale under the destination weighs the item; a camera checks the place.
Each stage succeeds most of the time, so why spend time on step 5? The simulation below compares two versions of one illustrative suction cell, over 5000 order lines: one that trusts every pick, and one that checks the vacuum after gripping and again at the drop-off, retrying up to three times.
import numpy as np
rng = np.random.default_rng(2026)
# Stage times (s) and success probabilities for one suction-gripper bin-picking cell (illustrative).
T_SEE, T_PLAN, T_GRASP, T_MOVE, T_CHECK = 0.4, 0.3, 1.2, 1.5, 0.1
P_SEE, P_GRASP, P_HOLD = 0.97, 0.92, 0.98 # find a target; seal and lift; keep it in transit
def pick_one(verify, max_tries=3):
"""Time spent and whether the item really arrived, for one order line."""
t = 0.0
for _ in range(max_tries if verify else 1):
t += T_SEE
while rng.random() > P_SEE: # nothing pickable seen: look again
t += T_SEE
t += T_PLAN + T_GRASP
grasped = rng.random() < P_GRASP
if verify:
t += T_CHECK # vacuum sensor: is something on the cup?
if not grasped:
continue # retry at once, without moving
held = grasped and rng.random() < P_HOLD
t += T_MOVE
if verify:
t += T_CHECK # still holding at the drop-off?
if held:
return t, True
continue # dropped on the way: try again
return t, held # no checks: the robot never knows
return t, False
for verify in (False, True):
results = [pick_one(verify) for _ in range(5000)]
total = sum(t for t, _ in results)
good = sum(ok for _, ok in results)
print(f"{'with checks + retries' if verify else 'no verification':22}: mean {total / len(results):.2f} s per line, "
f"{100 * good / len(results):6.2f}% delivered, {3600 * good / total:6.0f} good picks per hour, "
f"{len(results) - good} lines wrong")The cell without checks is slightly faster per line (3.41 s against 3.85 s) and delivers almost the same number of good picks per hour (948 against 935). But 506 of its 5000 order lines are silently wrong, about one in ten, each discovered later by a person packing an incomplete order or, worse, by a customer. With two 0.1 s vacuum checks and retries, only 1 line in 5000 fails, and that one is known to have failed, so it can be sent to a person. In logistics, a detected failure is cheap and an undetected one is expensive; verification is what turns a 92% grasp into a 99.98% process.
Learning to grasp, and grasping in industry
The analytic approach of this lesson needs a model of the object: its shape, its contacts, its friction. For a warehouse with millions of different products, most never seen before, that model does not exist. Since the mid-2010s the dominant approach has been to learn which grasps work, directly from images or point clouds.
- Learning from real attempts. Pinto and Gupta at Carnegie Mellon (2016) trained a network on 50,000 grasp attempts collected by a robot over 700 hours. A Google team (Levine and colleagues, 2018) collected about 800,000 attempts on a group of up to 14 robots and learned closed-loop grasping from camera images; QT-Opt (Kalashnikov and colleagues, 2018) used reinforcement learning on similar data.
- Learning from simulation and analytic labels. Dex-Net 2.0 (Mahler and colleagues at Berkeley, 2017) labelled millions of synthetic depth images with a robust version of the analytic quality you computed above, and trained a network to predict it for new objects.
- Grasps in clutter, in full 3-D. Later systems predict complete 6-D gripper poses directly from a scene's point cloud, among them 6-DOF GraspNet and Contact-GraspNet from NVIDIA's robotics lab, and the GraspNet-1Billion benchmark and AnyGrasp from Shanghai Jiao Tong University.
- Generalist policies. In 2025–2026, vision-language-action models trained on large multi-robot datasets treat grasping as one skill among many, chosen by language ("pick up the blue mug by its handle"). They are covered in Vision-language-action models; for high-throughput picking of a single kind of task, specialised grasp predictors remain common in 2026.
Bin picking is now a mature industry, with suppliers in many countries: Amazon Robotics (United States) deploys picking robots such as Sparrow, announced in 2022, in its own warehouses; Ocado Technology (United Kingdom) builds picking arms for automated grocery warehouses; Mujin (Japan) sells controllers for picking and depalletising; Fizyr (Netherlands) and Mech-Mind Robotics (China) supply the vision and grasp-detection software that many integrators use. Typical cells combine suction (often several cups of different sizes, swapped automatically) with careful verification, exception handling that hands awkward items to people, and continuous retraining on the failures they log.
Common misconceptions
Misconception
"Squeeze harder and any grasp will hold." Squeezing harder raises the friction available, but if the line between two fingers lies outside the friction cones, the squeeze itself pushes the object out sideways, as when a wet bar of soap shoots out of your hand. Force closure is about geometry and , not grip strength.
Misconception
"Force closure means the grasp is good." It means the grasp can resist every wrench in principle. A grasp with a tiny quality, like the 15° grasp of the oval, resists only small disturbances for a given grip force and fails with small errors in contact position or friction. Robust grasp planning maximises quality under the expected uncertainty.
Misconception
"Suction is a universal gripper." It needs a sealable surface and enough area for the load. Porous fabric, perforated boxes, very curved small items and dusty surfaces defeat it, which is why logistics cells keep jaws or people for the exceptions.
Misconception
"If the robot completed the motion, it delivered the item." Without a check, an empty gripper completes the motion just as smoothly. The pipeline simulation shows the cost: one order line in ten silently wrong.
A short history
Mechanicians studied how fixtures immobilise parts long before robots; Franz Reuleaux discussed closure of kinematic pairs in the 1870s. Robotic grasp analysis took its modern form in the 1980s: Kenneth Salisbury's work on articulated hands at Stanford and MIT, Matthew Mason's on the mechanics of pushing and grasping at MIT and Carnegie Mellon, and Van-Duc Nguyen's 1988 conditions for force-closure grasps. In 1992 Carlo Ferrari and John Canny at Berkeley defined the quality measure used above. Suction and parallel jaws dominated factories throughout; the shift to learned grasping came in the 2010s with deep learning, cheap depth cameras and large datasets, accelerated by the Amazon Picking Challenge and by the growth of online retail.
Exercises
1. Manipulation and grasping
Warm-upHumanA warehouse in Nairobi wants a robot to pick items from totes for online orders. Most items are sealed cardboard boxes and plastic bottles of many sizes, often lying in clutter with only one face visible. Which gripper is the natural first choice?
2. Manipulation and grasping
CoreComputationalA suction cup has an effective diameter of 40 mm, and the pump lowers the pressure inside it by 50 kPa below the surrounding air. What is the ideal holding force, in newtons (nearest whole number)?
3. Manipulation and grasping
CoreHumanWhat is the smallest number of frictionless point contacts that can hold a rigid object in force closure (form closure) in the plane?
4. Manipulation and grasping
CoreComputationalTwo fingers press on opposite parallel faces of a block, but they are offset, so the line joining the contacts makes an angle of 35° with the faces' normals. Which is the smallest friction coefficient in this list that makes the grasp force closure?
5. Manipulation and grasping
CoreHumanA suction picking robot must check that it really is holding the item before it moves on. Which of these signals carry real evidence? Select all that apply.
6. Manipulation and grasping
CoreComputationalWrite contact_wrenches(points, normals, mu, rho) for planar grasps. For each contact point with inward normal (normalise it first) and tangent , produce the friction-cone edge forces and (or just when mu == 0), and for each one the wrench row . Return all rows as a NumPy array, in contact order, edge first.
7. Manipulation and grasping
StretchComputationalWrite two_finger_closure(p1, n1, p2, n2, mu) using Nguyen's condition: a planar two-finger grasp with friction is force closure exactly when the segment from p1 to p2 makes an angle strictly less than with the inward normal n1, and the segment from p2 to p1 makes an angle strictly less than with n2. Normals may have any length. Return a Python bool. The tests include the lesson's oval, where your function must agree with the convex-hull test.
Further reading and resources
- Modern Robotics: Mechanics, Planning, and Control · Kevin M. Lynch and Frank C. Park · 2017Chapter 12 treats grasping and manipulation, including form and force closure; the preprint is free.
- Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics · Jeffrey Mahler and colleagues · 2017Training a grasp-quality network on millions of synthetic examples labelled by analytic metrics.
- Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours · Lerrel Pinto and Abhinav Gupta · 2016Learning grasps from a robot's own trial and error.
- Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection · Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz and Deirdre Quillen · 2018Hundreds of thousands of grasp attempts collected on a group of real robots.
- Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes · Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel and Dieter Fox · 2021Predicting full 6-D grasps directly from a depth image of clutter.
- GraspNetThe GraspNet-1Billion benchmark and dataset from Shanghai Jiao Tong University.
- MoveItThe open-source motion-planning framework for ROS used in many pick-and-place systems.