Skip to content
LumenaryAcademy
Frontier 75 min

Conformal prediction

Prediction intervals and prediction sets that hold their promised coverage for any model and any distribution, with one assumption — exchangeability — and what to do when it fails.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

A promise about the next case

A team in Nairobi has trained a gradient-boosted model that predicts the yield of a maize plot from satellite and weather features. A team in Lyon has a neural network that reads a skin image and names one of seven conditions. A team in Seoul has a language model that answers questions about tax rules. All three produce a single best guess, and all three are asked the same question by the people who will use them: how wrong could this be?

The classical answers need a model you believe. A regression interval y^±1.96 σ^\hat y \pm 1.96\,\hat\sigma relies on normal, constant-variance errors and a correctly specified mean. A Bayesian posterior predictive needs a prior and a likelihood. A boosted tree or a transformer has neither, and its own "confidence" scores are often badly calibrated.

Conformal prediction gives a different kind of answer. Wrap any trained model in a short calibration step, and it returns, for each new input xx, a set C(x)C(x) — an interval for a number, a set of labels for a class — with the guarantee

Pr⁡(Ynew∈C(Xnew))≥1−α\Pr\big(Y_{\text{new}} \in C(X_{\text{new}})\big) \ge 1 - \alpha

for a level you choose, such as 90%. The guarantee holds in finite samples, for any model however poor, and for any data distribution, under a single assumption: the calibration data and the new case are exchangeable. Vladimir Vovk, Alexander Gammerman and Glenn Shafer developed the idea from the late 1990s (their monograph is Algorithmic Learning in a Random World; the standard modern tutorial is by Angelopoulos and Bates), and since about 2019 it has spread through machine learning, medicine and the engineering of language-model systems.

IntuitionH

The rank argument

Everything in conformal prediction rests on one observation about ranks.

Suppose we have a model μ^\hat\mu, already trained, and a fresh calibration set of nn labelled examples it has never seen. For each one, compute a nonconformity score that says how badly the example fits the model — for regression, the absolute residual si=∣yi−μ^(xi)∣s_i = |y_i - \hat\mu(x_i)|. Now a new example arrives, with score sn+1s_{n+1} that we cannot compute yet because we do not know its label.

If the n+1n + 1 examples are exchangeable, their scores are too, and so the new score is equally likely to be the smallest, the second smallest, … or the largest of the n+1n + 1. Its rank is uniform. So the chance that it falls at or below the kk-th smallest calibration score is exactly k/(n+1)k/(n+1) (when there are no ties). Pick kk so that this is at least 1−α1 - \alpha, call that score q^\hat q, and the set of labels whose score would be at most q^\hat q — for regression, the interval μ^(x)±q^\hat\mu(x) \pm \hat q — covers the truth with probability at least 1−α1 - \alpha.

q̂ = 8th smallest of the 9 calibration scores12345678910the new score lands at or below q̂ in 8 of the 10 equally likely gapsso the probability of coverage is 8/10 = 1 − αscore (larger = stranger)
The rank argument with n = 9 calibration scores and α = 0.2. The new example's score is equally likely to fall into any of the 10 gaps. Taking q̂ as the ⌈10 × 0.8⌉ = 8th smallest calibration score, the new score is at most q̂ in 8 of the 10 gaps.

No model of the data appears anywhere in the argument. That is the source of conformal prediction's generality, and also of its limits: the guarantee is about the rank of a score among exchangeable scores, and says nothing about any particular xx.

DefinitionsH

Split conformal prediction

Definition— Exchangeability

Random variables Z1,…,ZmZ_1, \dots, Z_m are exchangeable if their joint distribution is unchanged by any permutation: (Zπ(1),…,Zπ(m))(Z_{\pi(1)}, \dots, Z_{\pi(m)}) has the same distribution as (Z1,…,Zm)(Z_1, \dots, Z_m) for every permutation π\pi. Independent, identically distributed variables are exchangeable; so are draws without replacement from a finite population. A time series with a trend is not.

Definition— Split conformal prediction

Given a trained model, a score function s(x,y)s(x, y) (large means the pair fits the model badly), a calibration set (x1,y1),…,(xn,yn)(x_1, y_1), \dots, (x_n, y_n) not used in training, and a level α\alpha:

  1. compute the calibration scores si=s(xi,yi)s_i = s(x_i, y_i);
  2. let k=⌈(n+1)(1−α)⌉k = \lceil (n + 1)(1 - \alpha) \rceil and let q^\hat q be the kk-th smallest score (or +∞+\infty if k>nk > n);
  3. for a new input xx, output C(x)={ y:s(x,y)≤q^ }C(x) = \{\, y : s(x, y) \le \hat q \,\}.

Theorem— Marginal coverage of split conformal prediction

If the calibration examples and the new example (Xn+1,Yn+1)(X_{n+1}, Y_{n+1}) are exchangeable, then

Pr⁡(Yn+1∈C(Xn+1))≥1−α.\Pr\big(Y_{n+1} \in C(X_{n+1})\big) \ge 1 - \alpha .

If, in addition, the scores are almost surely distinct, then the coverage is also at most 1−α+1n+11 - \alpha + \frac{1}{n+1}.

Proof

The model was trained on separate data, so it is fixed, and the scores s1,…,sn+1s_1, \dots, s_{n+1} are exchangeable. Assume first that they are distinct. Then Yn+1∈C(Xn+1)Y_{n+1} \in C(X_{n+1}) exactly when sn+1≤q^s_{n+1} \le \hat q, that is, when at most k−1k - 1 calibration scores lie below sn+1s_{n+1}: when the rank of sn+1s_{n+1} among all n+1n + 1 scores is at most kk. By exchangeability every rank from 11 to n+1n + 1 is equally likely, so the probability is k/(n+1)k/(n+1). Since (n+1)(1−α)≤k<(n+1)(1−α)+1(n+1)(1-\alpha) \le k < (n+1)(1-\alpha) + 1, this lies between 1−α1 - \alpha and 1−α+1n+11 - \alpha + \frac1{n+1}. Ties can only make sn+1≤q^s_{n+1} \le \hat q more likely, so the lower bound survives them.

∎

Three remarks matter in practice. The n+1n + 1 is not pedantry: with 19 calibration points and α=0.05\alpha = 0.05, k=⌈20×0.95⌉=19k = \lceil 20 \times 0.95 \rceil = 19, the largest score; with 18 points, k=19>18k = 19 > 18 and the honest interval is infinite. Using the plain empirical 95% quantile of the calibration scores would quietly under-cover. Second, the score is where the modelling goes: any function will give valid coverage, but a good one gives small sets. Third, "split" means the calibration data must not have been used for training — reuse them and the scores are no longer exchangeable with a new one, because the model has seen them. Full conformal prediction avoids the split by refitting the model once for every candidate label, at great computational cost, and the jackknife+ and CV+ methods of Barber, Candès, Ramdas and Tibshirani sit in between, using leave-one-out or cross-validation fits with a slightly weaker guarantee.

ComputationC

Split conformal regression, checked

Here is the whole procedure on synthetic data whose noise grows with xx: train a model, calibrate on 500 fresh points, and check the coverage. Because we built the world, we can compute the exact coverage of any interval by integrating over the known noise distribution, with no test-set noise in the check.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(2026)
mean_fn = lambda x: 2 + 0.8 * x + 0.1 * x**2 * np.sin(x)    # the true regression function
sd_fn = lambda x: 0.3 + 0.25 * x                             # noise grows with x

def simulate(m):
    x = rng.uniform(0, 10, m)
    return x, mean_fn(x) + rng.normal(0, sd_fn(x))

# 1. Fit any model on a training set. Here: a degree-7 polynomial by least squares.
x_tr, y_tr = simulate(1000)
coef = np.polyfit(x_tr, y_tr, deg=7)
predict = lambda x: np.polyval(coef, x)

# 2. Calibrate on fresh data. The score is the absolute residual.
alpha, n_cal = 0.10, 500
x_cal, y_cal = simulate(n_cal)
scores = np.abs(y_cal - predict(x_cal))
k = int(np.ceil((n_cal + 1) * (1 - alpha)))       # use the k-th smallest score
q_hat = np.sort(scores)[k - 1]
print(f"k = {k} of n = {n_cal}; q_hat = {q_hat:.3f}, so the interval is prediction +/- {q_hat:.3f}")

# 3. Because we built the world, we can compute the exact coverage of any interval width q.
grid = np.linspace(0, 10, 4001)
def true_coverage(q):
    err = mean_fn(grid) - predict(grid)
    return np.mean(stats.norm.cdf((err + q) / sd_fn(grid)) - stats.norm.cdf((err - q) / sd_fn(grid)))
print(f"exact coverage of this interval: {true_coverage(q_hat):.4f} (target: at least {1 - alpha:.2f})")

# 4. The coverage depends on the calibration set, which is random. Recalibrate 2,000 times.
cov = np.array([true_coverage(np.sort(np.abs(yc - predict(xc)))[k - 1])
                for xc, yc in (simulate(n_cal) for _ in range(2000))])
theory = stats.beta(k, n_cal + 1 - k)               # the distribution conformal theory predicts
print(f"\nover 2,000 calibration sets: mean {cov.mean():.4f}, sd {cov.std():.4f}, "
      f"5th-95th percentile {np.quantile(cov, 0.05):.3f}-{np.quantile(cov, 0.95):.3f}")
print(f"theory, Beta({k}, {n_cal + 1 - k}):       mean {theory.mean():.4f}, sd {theory.std():.4f}, "
      f"5th-95th percentile {theory.ppf(0.05):.3f}-{theory.ppf(0.95):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The theory behind the last line is worth knowing. Given the calibration set, the coverage of the interval is F(q^)F(\hat q), where FF is the distribution function of the score. Since F(s1),…,F(sn)F(s_1), \dots, F(s_n) are independent uniforms and q^\hat q is the kk-th order statistic, the coverage has a Beta(k,n+1−k)\mathrm{Beta}(k, n + 1 - k) distribution (Vovk, 2012). The guarantee "at least 90%" is an average over calibration sets; any one calibration set may give a little less. If you need a guarantee for your calibration set — "with probability 95%, my coverage is at least 88%" — you can read it off the Beta distribution and choose a slightly larger kk. With 500 calibration points the wobble is about ±1.3\pm 1.3 percentage points; with 5,000 it is about ±0.4\pm 0.4.

ExplanationH

Marginal is not conditional

The guarantee averages over new inputs as well as over calibration sets. It says that 90% of cases are covered, not that each case has a 90% chance. When the noise varies with xx, an interval of constant width is too wide where the data are quiet and too narrow where they are noisy, and the average can still be exactly right.

The fix is in the score. Divide the residual by an estimate of the local spread, s(x,y)=∣y−μ^(x)∣/σ^(x)s(x, y) = |y - \hat\mu(x)| / \hat\sigma(x), where σ^\hat\sigma is a second model fitted on the training data; the interval becomes μ^(x)±q^ σ^(x)\hat\mu(x) \pm \hat q\, \hat\sigma(x). The coverage guarantee is unchanged — it never depended on the score — but the width now follows the noise.

python
import numpy as np

rng = np.random.default_rng(7)
mean_fn = lambda x: 2 + 0.8 * x + 0.1 * x**2 * np.sin(x)
sd_fn = lambda x: 0.3 + 0.25 * x

def simulate(m):
    x = rng.uniform(0, 10, m)
    return x, mean_fn(x) + rng.normal(0, sd_fn(x))

def conformal_quantile(scores, alpha):
    k = int(np.ceil((scores.size + 1) * (1 - alpha)))
    return np.sort(scores)[k - 1]

alpha = 0.10
x_tr, y_tr = simulate(1000)
coef = np.polyfit(x_tr, y_tr, 7)
mu = lambda x: np.polyval(coef, x)                       # model for the mean
coef_s = np.polyfit(x_tr, np.abs(y_tr - mu(x_tr)), 1)   # second model, for the spread: |residual| on x
spread = lambda x: np.maximum(np.polyval(coef_s, x), 0.05)

x_cal, y_cal = simulate(1000)
q_abs = conformal_quantile(np.abs(y_cal - mu(x_cal)), alpha)                   # score 1: |residual|
q_norm = conformal_quantile(np.abs(y_cal - mu(x_cal)) / spread(x_cal), alpha)  # score 2: |residual| / spread

x_te, y_te = simulate(50000)
res = np.abs(y_te - mu(x_te))
cover_abs, width_abs = res <= q_abs, np.full(x_te.size, 2 * q_abs)
cover_norm, width_norm = res <= q_norm * spread(x_te), 2 * q_norm * spread(x_te)

print(f"{'x range':>9} | {'|residual| score':^21} | {'normalised score':^21}")
print(f"{'':>9} | {'coverage':>9} {'width':>10} | {'coverage':>9} {'width':>10}")
for lo in range(0, 10, 2):
    b = (x_te >= lo) & (x_te < lo + 2)
    print(f"{lo:>4}-{lo + 2:<4} | {cover_abs[b].mean():9.3f} {width_abs[b].mean():10.2f} | "
          f"{cover_norm[b].mean():9.3f} {width_norm[b].mean():10.2f}")
print(f"{'overall':>9} | {cover_abs.mean():9.3f} {width_abs.mean():10.2f} | "
      f"{cover_norm.mean():9.3f} {width_norm.mean():10.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
0246810input x01020outcome ynormalised score: width follows the noiseabsolute residual: constant width
One synthetic sample from the code's world (110 points; noise grows with x). The dashed curves are the constant-width conformal interval: 8 of the 9 points outside it have x > 6. The shaded band uses the normalised score, so it is narrow where the data are quiet and wide where they are noisy, with the same 90% marginal guarantee.

Both intervals have marginal coverage of about 90%, as promised. But the constant-width interval covers 100% of cases with xx below 2 and only 77% of those above 8, while the normalised interval stays between 0.89 and 0.92 in every band — and is narrower on average, 5.11 against 6.08. Other good scores exist: conformalized quantile regression (Romano, Patterson and Candès) fits models for the 5% and 95% conditional quantiles and conformalises the distance outside them; it is a widely used default.

ComputationC

Prediction sets for classification

For a classifier that outputs probabilities p^1(x),…,p^K(x)\hat p_1(x), \dots, \hat p_K(x), the simplest score is one minus the probability given to the true class, s(x,y)=1−p^y(x)s(x, y) = 1 - \hat p_y(x). The prediction set then contains every class whose probability is at least 1−q^1 - \hat q (Sadinle, Lei and Wasserman showed it gives the smallest sets on average among methods with the guarantee, if the probabilities are right). The model need not be calibrated or even good: conformal prediction turns whatever scores it produces into sets with the promised coverage.

The example classifies land cover — water, forest, cropland, urban — from two synthetic features of the kind derived from satellite images, with a deliberately crude classifier (Gaussian classes with a shared covariance, which the data do not have).

python
import numpy as np

rng = np.random.default_rng(42)
classes = ["water", "forest", "cropland", "urban"]
centres = np.array([[0.0, 0.0], [2.5, 0.5], [1.5, 2.2], [3.5, 2.8]])
scales = np.array([0.6, 1.0, 1.1, 0.8])               # some classes are more spread out than others

def simulate(m):
    y = rng.integers(0, 4, m)
    x = centres[y] + rng.normal(size=(m, 2)) * scales[y][:, None]
    return x, y

# A simple, deliberately imperfect classifier: Gaussians with one shared covariance (LDA).
x_tr, y_tr = simulate(2000)
means = np.array([x_tr[y_tr == c].mean(axis=0) for c in range(4)])
cov = sum(np.cov(x_tr[y_tr == c].T) for c in range(4)) / 4
P = np.linalg.inv(cov)
def predict_proba(x):
    logits = x @ P @ means.T - 0.5 * np.sum(means @ P * means, axis=1)
    logits -= logits.max(axis=1, keepdims=True)
    p = np.exp(logits)
    return p / p.sum(axis=1, keepdims=True)

alpha = 0.10
x_cal, y_cal = simulate(1000)
p_cal = predict_proba(x_cal)
scores = 1 - p_cal[np.arange(y_cal.size), y_cal]         # score: 1 - probability given to the true class
n = scores.size
q_hat = np.sort(scores)[int(np.ceil((n + 1) * (1 - alpha))) - 1]
print(f"q_hat = {q_hat:.3f}: a class is in the set when the model gives it probability >= {1 - q_hat:.3f}")

x_te, y_te = simulate(20000)
p_te = predict_proba(x_te)
sets = p_te >= 1 - q_hat                                  # boolean matrix: row i = prediction set for point i
covered = sets[np.arange(y_te.size), y_te]
size = sets.sum(axis=1)
top1 = (p_te.argmax(axis=1) == y_te).mean()
print(f"top-1 accuracy of the classifier: {top1:.3f}")
print(f"coverage of the 90% sets: {covered.mean():.3f};  average set size: {size.mean():.2f}")
print("distribution of set sizes:", {s: round(float(np.mean(size == s)), 3) for s in range(0, 5)})
print("\ncoverage by true class (only the average over classes is guaranteed):")
for c, name in enumerate(classes):
    print(f"  {name:9s} {covered[y_te == c].mean():.3f}   average set size {size[y_te == c].mean():.2f}")

# Two example inputs: one clear-cut, one ambiguous.
for x in [np.array([[-0.5, -0.4]]), np.array([[2.4, 1.6]])]:
    p = predict_proba(x)[0]
    chosen = [classes[c] for c in range(4) if p[c] >= 1 - q_hat]
    print(f"\nx = {x[0]}: probabilities {np.round(p, 2)} -> set {chosen}")

# Class-conditional ("Mondrian") calibration: one threshold per class, from that class's scores only.
q_class = np.array([np.sort(s)[int(np.ceil((s.size + 1) * (1 - alpha))) - 1]
                    for s in (scores[y_cal == c] for c in range(4))])
sets_c = p_te >= 1 - q_class[None, :]
cov_c = sets_c[np.arange(y_te.size), y_te]
print("\nclass-conditional calibration:")
for c, name in enumerate(classes):
    print(f"  {name:9s} {cov_c[y_te == c].mean():.3f}")
print(f"  average set size {sets_c.sum(axis=1).mean():.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The classifier alone is right 80.1% of the time; its 90% sets cover 91.1% of cases, with an average size of 1.38. The set size is the uncertainty signal: two-thirds of inputs get a single label, and the ambiguous input at (2.4,1.6)(2.4, 1.6), where three classes overlap, gets three. A downstream user can act on a singleton and send a large set for human review.

The guarantee is marginal over classes as well as inputs, and the run shows the price: water, which is easy, is covered 99.6% of the time, cropland only 81.2%. Calibrating separately within each class — class-conditional or Mondrian conformal prediction, named by Vovk and colleagues after Piet Mondrian's grid paintings because it partitions the data — gives each class its own guarantee. The four per-class coverages become 0.905, 0.913, 0.906 and 0.876; with about 250 calibration points per class, a wobble of ±2\pm 2 points is expected, as the Beta distribution of the previous section predicts. For many classes and images, the APS and RAPS scores (Romano, Sesia and Candès; Angelopoulos et al.) add the probabilities of classes in order until the true class is reached, which gives sets that adapt better to hard inputs.

ComputationC

When exchangeability fails

Exchangeability is a strong assumption in disguise. It fails when:

  • the inputs shift (covariate shift): a model calibrated on one hospital's patients is used at another, or a yield model calibrated in wet years meets a drought;
  • the labels shift: a disease becomes more common during an outbreak;
  • the data are ordered in time and the world drifts;
  • feedback links prediction and outcome, as when a model's predictions change who is treated.

Under covariate shift — the distribution of XX changes, but YY given XX does not — there is a repair. Tibshirani, Barber, Candès and Ramdas showed that weighting each calibration score by the likelihood ratio w(x)=pnew(x)/pcal(x)w(x) = p_{\text{new}}(x)/p_{\text{cal}}(x), and giving the test point's own weight to a score of +∞+\infty, restores the guarantee. The ratio is rarely known, but it can be estimated with a classifier trained to tell calibration inputs from unlabelled deployment inputs, which are usually easy to collect. Below, inputs are centred at 3 during training and calibration and at 5 in deployment, where the noise is larger.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(11)
mean_fn = lambda x: 2 + 0.8 * x + 0.1 * x**2 * np.sin(x)
sd_fn = lambda x: 0.3 + 0.25 * np.abs(x)

def simulate(m, centre):
    x = rng.normal(centre, 1.5, m)                     # where the inputs come from can shift
    return x, mean_fn(x) + rng.normal(0, sd_fn(x))

alpha = 0.10
x_tr, y_tr = simulate(2000, 3.0)                       # training population: inputs centred at 3
coef = np.polyfit(x_tr, y_tr, 7)
mu = lambda x: np.polyval(coef, x)

def split_quantile(s, alpha):
    return np.sort(s)[int(np.ceil((s.size + 1) * (1 - alpha))) - 1]

def weighted_quantiles(s_cal, w_cal, w_test, alpha):
    """Weighted conformal (Tibshirani et al., 2019): calibration scores weighted by the density
    ratio, plus a point mass at +infinity carrying the test point's own weight."""
    order = np.argsort(s_cal)
    s_sorted, cum_w = s_cal[order], np.cumsum(w_cal[order])
    need = (1 - alpha) * (cum_w[-1] + w_test)           # weight needed at or below the quantile
    j = np.searchsorted(cum_w, need)
    return np.where(j < s_sorted.size, s_sorted[np.minimum(j, s_sorted.size - 1)], np.inf)

def estimate_ratio(x_cal, x_unlabelled):
    """Logistic regression telling calibration inputs (0) from unlabelled deployment inputs (1).
    With equal sample sizes, its odds estimate the density ratio p_deploy(x) / p_cal(x)."""
    Z = np.column_stack([np.ones(x_cal.size + x_unlabelled.size), np.r_[x_cal, x_unlabelled]])
    t = np.r_[np.zeros(x_cal.size), np.ones(x_unlabelled.size)]
    w = np.zeros(2)
    for _ in range(25):                                 # Newton's method
        p = 1 / (1 + np.exp(-Z @ w))
        w += np.linalg.solve(Z.T @ (Z * (p * (1 - p))[:, None]), Z.T @ (t - p))
    return lambda x: np.exp(w[0] + w[1] * x)

true_ratio = lambda x: stats.norm.pdf(x, 5.0, 1.5) / stats.norm.pdf(x, 3.0, 1.5)
cov = {"ordinary split conformal": [], "weighted, true ratio": [], "weighted, estimated ratio": []}
for rep in range(200):
    x_cal, y_cal = simulate(1000, 3.0)                  # calibration: same population as training
    s_cal = np.abs(y_cal - mu(x_cal))
    x_te, y_te = simulate(2000, 5.0)                    # deployment: inputs have shifted upwards
    s_te = np.abs(y_te - mu(x_te))
    est_ratio = estimate_ratio(x_cal, rng.normal(5.0, 1.5, 1000))
    cov["ordinary split conformal"].append(np.mean(s_te <= split_quantile(s_cal, alpha)))
    cov["weighted, true ratio"].append(np.mean(s_te <= weighted_quantiles(s_cal, true_ratio(x_cal), true_ratio(x_te), alpha)))
    cov["weighted, estimated ratio"].append(np.mean(s_te <= weighted_quantiles(s_cal, est_ratio(x_cal), est_ratio(x_te), alpha)))

print("coverage in deployment, over 200 repetitions (target 0.90):")
for name, c in cov.items():
    print(f"  {name:26s} mean {np.mean(c):.3f}   middle 90% of runs {np.quantile(c, 0.05):.3f}-{np.quantile(c, 0.95):.3f}")
w = true_ratio(x_cal)
print(f"effective calibration sample size after weighting (last run): {w.sum() ** 2 / (w ** 2).sum():.0f} of 1000")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Ordinary conformal prediction, blind to the shift, covers only 76.3% of deployment cases. Weighting restores the average coverage to about 90.7%, and the estimated ratio does as well as the true one. But the runs vary much more, from 0.852 to 0.971: weighting concentrates the calibration on the few points that resemble deployment, leaving an effective sample size of about 169 out of 1,000. When the shift is so large that almost no calibration points resemble deployment, the honest weighted interval becomes infinite.

For data in time, adaptive conformal inference (Gibbs and Candès, 2021) adjusts α\alpha online — widening after misses and narrowing after hits — and guarantees that the long-run miss rate approaches α\alpha even under arbitrary drift, at the cost of saying less about any single time. Barber et al. bound how much coverage can be lost when exchangeability holds only approximately.

ExplanationH

Conformal prediction in practice (2026)

Medicine. Clinical uses are attractive because a set is a natural clinical object: a differential diagnosis, a range for a laboratory value, a flag that says "uncertain — refer". Researchers have built conformal sets for dermatology and radiology classifiers, intervals for drug-response and length-of-stay predictions, and, for more than a decade, conformal predictors for compound activity in pharmaceutical chemistry. The open problems are statistical, not computational. Marginal coverage across patients does not mean coverage for this patient, and coverage can be uneven across age groups, sexes or ethnic groups unless those groups are calibrated separately; models move between hospitals, scanners and years, breaking exchangeability; and the calibration labels themselves (a pathologist's diagnosis, a coded outcome) contain errors. As of 2026, conformal methods appear mainly in research studies and in some decision-support prototypes. Their coverage guarantees complement prospective clinical validation; they do not replace it.

Language models. A language model's output is not a number or a label, so the first task is to define what "covered" means. Several ideas are now established. Conformal language modelling (Quach et al., 2023) samples answers until the set of samples contains an acceptable one with a guaranteed probability, using a calibrated stopping rule. Conformal factuality (Mohri and Hashimoto, 2024) splits a long answer into individual claims and removes the least confident ones until the remaining answer is entirely correct with probability 1−α1 - \alpha. KnowNo (Ren et al., 2023) lets a robot's language-model planner form a conformal set of possible actions and ask a human for help when the set has more than one member. More generally, conformal risk control (Angelopoulos et al., 2022) and Learn then Test (Angelopoulos et al., 2021) calibrate thresholds to control other error rates, such as the fraction of missed tumour pixels in a segmentation or the rate of unsafe responses.

Each of these needs a labelled calibration set drawn from the prompts the system will actually face, and a definition of a correct or acceptable answer that someone has to supply and check. Prompt distributions drift quickly, so the exchangeability assumption needs particular care. The guarantees are real, and narrower than they sound in a press release.

Software. Conformal methods take a few lines of NumPy, as above. Open-source implementations include MAPIE in Python and the conformalInference package in R; deep-learning frameworks have their own community libraries. Whatever the tool, check that its calibration step uses data held out from training and applies the (n+1)(n+1) correction.

Common misconceptionsH

Common misconceptions

Misconception

"A 90% conformal interval contains this case's outcome with probability 0.9." The guarantee is marginal: averaged over new cases and over calibration sets. For a particular xx the coverage may be higher or lower — 100% or 77% in the bands of the second example. Conditional guarantees need group-wise calibration or extra assumptions.

Misconception

"Conformal prediction makes a bad model good." It makes a bad model honest. Coverage holds for any model, but a poor model has large scores and therefore wide intervals or large sets. Efficiency — small sets — still has to be earned by modelling.

Misconception

"Distribution-free means assumption-free." The method needs exchangeability between calibration and future data, and calibration data that the model was not trained on. Under shift or drift the guarantee can fail badly, as the 76% coverage above shows.

Misconception

"My calibration set gives exactly 90% coverage." Given the calibration set, coverage has a Beta distribution centred near 1−α1 - \alpha; with 500 points its standard deviation is about 1.3 percentage points. Use more calibration data, or a slightly more conservative kk, if you need a guarantee for the calibration set you actually have.

Exercises

Exercises

1. Conformal prediction

Warm-upHuman

You have n=99n = 99 calibration residuals and want a 90% split conformal interval (α=0.1\alpha = 0.1). The half-width q^\hat q is the kk-th smallest residual. What is kk?

2. Conformal prediction

CoreHuman

For a 95% split conformal interval (α=0.05\alpha = 0.05), what is the smallest number of calibration points nn for which the interval is finite, that is, for which ⌈(n+1)(1−α)⌉≤n\lceil (n+1)(1-\alpha) \rceil \le n?

3. Conformal prediction

CoreHuman

A hospital calibrates a 90% split conformal interval for patients' length of stay on 2,000 recent discharges from the same hospital. Which statement is justified by the conformal guarantee alone?

Choose one

4. Conformal prediction

CoreHuman

In which situations does the split conformal guarantee not apply as stated? Select all that apply.

Choose all that apply

5. Conformal prediction

CoreComputational

Write conformal_halfwidth(residuals, alpha) that returns the split conformal half-width q^\hat q from an array of absolute calibration residuals: the kk-th smallest residual with k=⌈(n+1)(1−α)⌉k = \lceil (n+1)(1-\alpha) \rceil, or np.inf if k>nk > n.

The starter uses the plain empirical quantile, a common mistake; the tests will show why it is not enough.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

6. Conformal prediction

StretchComputational

Write prediction_sets(p_cal, y_cal, p_test, alpha) for conformal classification with the score 1−p^y(x)1 - \hat p_y(x).

  • p_cal: array of shape (n, K), the model's class probabilities on the calibration inputs;
  • y_cal: integer array of shape (n,), their true classes;
  • p_test: array of shape (m, K).

Return a boolean array of shape (m, K) whose entry [i, c] is True when class c belongs to the prediction set of test input i. If k=⌈(n+1)(1−α)⌉>nk = \lceil (n+1)(1-\alpha) \rceil > n, every class belongs to every set.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Conformal prediction

Warm-upHuman

Conformal prediction's coverage guarantee needs only one assumption about the data: that the calibration examples and the new example are ______ (one word).

Further reading and resources