Skip to content
LumenaryAcademy
Advanced 80 min

High-dimensional statistics

When there are more variables than observations — ridge and the lasso, the bias–variance trade-off, principal components, thousands of tests at once, and the problem of inference after selection.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

More variables than observations

A cancer research group measures the activity of 20,000 genes in tumour samples from 100 patients, and wants to know which genes predict whether a tumour responds to a drug. A neuroscientist records 50,000 voxels of brain activity in 30 volunteers. A bank tracks 2,000 possible risk factors for a few hundred defaulted loans. In each case the number of variables pp is larger, often far larger, than the number of observations nn.

Classical statistics was built for the opposite situation: a handful of carefully chosen variables and many observations. When p>np > n, its central tool breaks. Least squares, from Multiple regression and diagnostics, has infinitely many solutions that fit the data perfectly, each with a residual of exactly zero, and each predicting new cases differently. Tests designed for one hypothesis are applied to 20,000 at once, and at the 5% level about a thousand of the "discoveries" will be noise. And choosing which variables to keep, then reporting their pp-values as if they had been chosen in advance, gives results that look far more certain than they are.

High-dimensional statistics is the set of ideas that make learning possible anyway. Its central assumption is that the problem has less structure than its size suggests: only a few genes matter (sparsity), or the many variables move together along a few hidden directions (low rank). Its central tools, penalised regression, principal components and error-rate control across many tests, are now standard in genomics, imaging, finance and machine learning.

IntuitionH

Why least squares breaks, and what a penalty buys

Least squares solves the normal equations X⊤Xβ=X⊤yX^\top X \beta = X^\top y. When p>np > n, the p×pp \times p matrix X⊤XX^\top X has rank at most nn, so it cannot be inverted: there is a whole subspace of coefficient vectors that fit the training data exactly. The usual software answer is the minimum-norm solution, the perfect fit with the smallest ∥β∥2\|\beta\|_2. It fits the training data perfectly and generalises badly, because a perfect fit to nn noisy points is mostly a fit to the noise.

The cure is to add a penalty that prefers small coefficients. Ridge regression minimises ∥y−Xβ∥22+λ∥β∥22,\|y - X\beta\|_2^2 + \lambda\|\beta\|_2^2, and for any λ>0\lambda \gt 0 the matrix X⊤X+λIX^\top X + \lambda I is invertible, so the solution is unique: β^λ=(X⊤X+λI)−1X⊤y\hat\beta_\lambda = (X^\top X + \lambda I)^{-1} X^\top y. The penalty adds bias, pulling estimates towards zero, and removes variance, because the estimates no longer chase every wobble of the noise. The bias–variance trade-off says the expected test error at a new point is bias2+variance+σ2,\text{bias}^2 + \text{variance} + \sigma^2, and λ\lambda moves us along it.

The code makes this concrete with n=90n = 90 observations, p=100p = 100 predictors and many small true effects. It fits ridge for a range of λ\lambda, first on one data set, then on 300 fresh training sets to measure the bias and variance of the predictions at 500 fixed test points.

python
import numpy as np

rng = np.random.default_rng(1970)
n, p, sigma = 90, 100, 1.0                              # more predictors than observations
beta = rng.normal(0, 0.3, p)                           # many small effects
X_test = rng.normal(size=(500, p))
f_test = X_test @ beta                                  # the true signal at 500 new points

def ridge_fits(X, y, lams):
    """Ridge coefficients for every lambda from one SVD. lambda = 0 gives the minimum-norm least-squares fit."""
    U, d, Vt = np.linalg.svd(X, full_matrices=False)
    uty = U.T @ y
    return [Vt.T @ (d / (d ** 2 + lam) * uty) for lam in lams]

lams = [0.0, 1.0, 5.0, 10.0, 20.0, 50.0, 100.0, 300.0, 1000.0]

# One data set: least squares fits the training data perfectly and predicts badly.
X = rng.normal(size=(n, p))
y = X @ beta + rng.normal(0, sigma, n)
print(" lambda   training MSE   test MSE")
for lam, b in zip(lams, ridge_fits(X, y, lams)):
    print(f"{lam:7.0f}   {np.mean((y - X @ b) ** 2):10.2f}   {np.mean((f_test + rng.normal(0, sigma, 500) - X_test @ b) ** 2):8.2f}")

# Bias and variance: refit on 300 fresh training sets and look at the predictions at the same test points.
reps = 300
preds = np.zeros((len(lams), reps, 500))
for r in range(reps):
    X = rng.normal(size=(n, p))
    y = X @ beta + rng.normal(0, sigma, n)
    for k, b in enumerate(ridge_fits(X, y, lams)):
        preds[k, r] = X_test @ b
bias2 = np.mean((preds.mean(axis=1) - f_test) ** 2, axis=1)
var = np.mean(preds.var(axis=1), axis=1)
print("\n lambda   bias^2   variance   bias^2 + variance + noise")
for lam, b2, v in zip(lams, bias2, var):
    print(f"{lam:7.0f}  {b2:7.2f}  {v:9.2f}   {b2 + v + sigma ** 2:9.2f}")

print(f"\npredicting 0 for everyone: expected test MSE {np.mean(f_test ** 2) + sigma ** 2:.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With λ=0\lambda = 0 the training error is 0.00 and the test error 9.37: the minimum-norm fit is perfect on the data it has seen and worse than useless on new data. Averaged over 300 training sets its expected test error is 11.96, higher than the 10.99 we would get by predicting zero for everyone. Its problem is almost pure variance (10.81). Increasing λ\lambda trades variance for bias, and the total is smallest, 4.18, at λ=10\lambda = 10, where the squared bias is 1.00 and the variance 2.18. Push λ\lambda to 1,000 and the model is nearly all bias.

11010010000246810noise level σ² = 1bias²variancetest errorbest: 4.18ridge penalty λ (logarithmic scale)
The bias–variance trade-off for ridge regression in the simulation above (n = 90, p = 100). Squared bias rises and variance falls as the penalty grows; the expected test error, their sum plus the noise, is lowest near λ = 10. At λ = 0, off the left of the chart, the variance is 10.81 and the test error 11.96.
DefinitionsH

Ridge, the lasso and the elastic net

Definition— ridge, lasso and elastic net

With predictors standardised to mean 0 and variance 1 and the response centred:

  • ridge regression minimises ∥y−Xβ∥22+λ∑jβj2\|y - X\beta\|_2^2 + \lambda\sum_j \beta_j^2;
  • the lasso (least absolute shrinkage and selection operator) minimises 12n∥y−Xβ∥22+λ∑j∣βj∣\frac{1}{2n}\|y - X\beta\|_2^2 + \lambda\sum_j |\beta_j|;
  • the elastic net uses a mixture of the two penalties, λ(α∑j∣βj∣+1−α2∑jβj2)\lambda\big(\alpha\sum_j|\beta_j| + \frac{1-\alpha}{2}\sum_j \beta_j^2\big).

Standardising matters, because the penalty treats all coefficients alike: without it, a predictor measured in millimetres would be penalised a thousand times less than the same predictor measured in metres. Different software scales the loss differently (with or without the 12n\frac{1}{2n}), so a value of λ\lambda means nothing without its convention.

Theorem— Ridge regression shrinks along principal directions

Let X=UDV⊤X = UDV^\top be the thin singular value decomposition, with singular values d1≥d2≥⋯>0d_1 \ge d_2 \ge \cdots \gt 0. For λ>0\lambda \gt 0 the ridge estimate is β^λ=V diag ⁣(djdj2+λ)U⊤y,Xβ^λ=∑jdj2dj2+λ ujuj⊤y.\hat\beta_\lambda = V\,\text{diag}\!\left(\frac{d_j}{d_j^2 + \lambda}\right) U^\top y, \qquad X\hat\beta_\lambda = \sum_j \frac{d_j^2}{d_j^2 + \lambda}\, u_j u_j^\top y. So the fitted values are the least-squares fit's components along the directions uju_j, each multiplied by a factor dj2/(dj2+λ)d_j^2/(d_j^2 + \lambda) between 0 and 1: directions in which the data vary little (small djd_j) are shrunk the most.

Proof

The ridge objective is strictly convex for λ>0\lambda \gt 0, so its unique minimiser solves (X⊤X+λI)β=X⊤y(X^\top X + \lambda I)\beta = X^\top y. Substitute the candidate β=VD(D2+λI)−1U⊤y\beta = V D (D^2 + \lambda I)^{-1} U^\top y. Since V⊤V=IV^\top V = I, we get X⊤Xβ=VD2V⊤VD(D2+λI)−1U⊤y=VD3(D2+λI)−1U⊤yX^\top X \beta = V D^2 V^\top V D (D^2 + \lambda I)^{-1} U^\top y = V D^3 (D^2 + \lambda I)^{-1} U^\top y, and adding λβ\lambda\beta gives VD(D2+λI)(D2+λI)−1U⊤y=VDU⊤y=X⊤yV D (D^2 + \lambda I)(D^2 + \lambda I)^{-1} U^\top y = V D U^\top y = X^\top y. Multiplying the candidate by X=UDV⊤X = UDV^\top gives the fitted values.

∎

This is exactly the formula in the code's ridge_fits, and it computes the whole path of λ\lambda values from one decomposition. Letting λ→0\lambda \to 0 gives factors 1/dj1/d_j, the minimum-norm least-squares solution, which is why tiny singular values made it explode.

The lasso behaves differently. When the columns of XX are orthogonal (so that X⊤X/n=IX^\top X/n = I), its solution is the least-squares estimate bjb_j passed through soft thresholding: β^j=sign(bj) (∣bj∣−λ)+,\hat\beta_j = \text{sign}(b_j)\,\big(|b_j| - \lambda\big)_+, where (z)+=max⁡(z,0)(z)_+ = \max(z, 0). Every coefficient is moved towards zero by λ\lambda, and those smaller than λ\lambda in absolute value become exactly zero. Ridge, by contrast, divides each coefficient by the same factor and never sets one to zero. So the lasso selects variables as well as shrinking them. The geometry shows why.

β₁β₂lasso: |β₁| + |β₂| ≤ tleast-squares estimateβ₁β₂ridge: β₁² + β₂² ≤ t²touches at a corner: β₂ = 0touches a smooth edge: both non-zero
Penalised regression as constrained least squares, for two coefficients. The ellipses are contours of the squared error around the least-squares estimate; the solution is where the smallest ellipse first touches the constraint region. The lasso's diamond has corners on the axes, where one coefficient is exactly zero; ridge's circle has none.

In Bayesian terms (Bayesian inference), ridge is the posterior mode under independent normal priors on the coefficients, and the lasso is the posterior mode under Laplace (double-exponential) priors, whose sharp peak at zero is what produces exact zeros. The elastic net was introduced because the lasso behaves erratically with groups of highly correlated predictors, tending to pick one arbitrarily; adding some ridge penalty makes it share the weight among them.

ComputationC

The lasso by coordinate descent

The lasso has no closed form when columns are correlated, but it has a beautifully simple algorithm. Coordinate descent cycles through the coefficients, and for each one solves the one-dimensional problem exactly with the others held fixed. That one-dimensional problem is a soft-thresholding step: compute the correlation of xjx_j with the current partial residual (the residual with xjx_j's own contribution added back), then shrink it by λ\lambda. Jerome Friedman, Trevor Hastie and Robert Tibshirani showed in 2010 that this, with warm starts along a decreasing sequence of λ\lambda values, computes whole lasso paths faster than earlier methods; their glmnet package is built on it.

python
import numpy as np

rng = np.random.default_rng(1996)
n, p = 100, 500
X = rng.normal(size=(n, p))
X = (X - X.mean(axis=0)) / X.std(axis=0)                # standardise: the penalty treats all columns alike
beta = np.zeros(p)
beta[:8] = [3.0, -2.5, 2.0, -1.5, 1.5, 1.0, -1.0, 0.8]  # only 8 of 500 predictors matter
y = X @ beta + rng.normal(0, 1.0, n)
y = y - y.mean()

def soft_threshold(z, t):
    return np.sign(z) * max(abs(z) - t, 0.0)

def lasso_cd(X, y, lam, b=None, sweeps=200, tol=1e-6):
    """Minimise (1/2n)||y - Xb||^2 + lam * ||b||_1 by cyclic coordinate descent (columns standardised)."""
    n, p = X.shape
    b = np.zeros(p) if b is None else b.copy()
    r = y - X @ b                                       # current residual
    for _ in range(sweeps):
        biggest = 0.0
        for j in range(p):
            old = b[j]
            rho = X[:, j] @ r / n + old                 # x_j'x_j / n = 1 after standardising
            b[j] = soft_threshold(rho, lam)
            if b[j] != old:
                r -= X[:, j] * (b[j] - old)
                biggest = max(biggest, abs(b[j] - old))
        if biggest < tol:
            break
    return b

lam_max = np.max(np.abs(X.T @ y)) / n                   # the smallest lambda at which every coefficient is 0
print(f"lambda_max = {lam_max:.3f}")
print("  lambda   nonzero   true found (of 8)   false positives   training R^2")
b = np.zeros(p)
for frac in [0.9, 0.6, 0.4, 0.25, 0.15, 0.08, 0.04, 0.02]:
    lam = frac * lam_max
    b = lasso_cd(X, y, lam, b)                          # warm start from the previous solution
    nz = np.nonzero(b)[0]
    r2 = 1 - np.sum((y - X @ b) ** 2) / np.sum(y ** 2)
    print(f"{lam:8.3f}   {nz.size:7d}   {np.sum(nz < 8):17d}   {np.sum(nz >= 8):15d}   {r2:12.3f}")

lam = 0.08 * lam_max
b = lasso_cd(X, y, lam, b)
print(f"estimates of the 8 real effects at lambda = {lam:.3f}:", np.round(b[:8], 2))
# Optimality (KKT) check: |x_j'r|/n = lam where b_j != 0, and <= lam where b_j = 0.
g = X.T @ (y - X @ b) / n
print(f"KKT check: max |gradient| on zero coefficients {np.max(np.abs(g[b == 0])):.4f} (should be <= {lam:.4f}); "
      f"on nonzero ones, |gradient| - lambda within {np.max(np.abs(np.abs(g[b != 0]) - lam)):.1e}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Reading the path from large λ\lambda to small: at λ=3.033\lambda = 3.033 only the strongest predictor enters; by λ=0.843\lambda = 0.843 all eight real effects are in and nothing else, a perfect selection. Below that, noise variables flood in: 6 false positives at 0.270, 60 at 0.067, where the training R2R^2 of 0.994 is a sign of overfitting, not success. At λ=0.270\lambda = 0.270 the eight real estimates are visibly shrunk: 2.78 for a true 3.0, 0.47 for a true 0.8. The last line checks the optimality (Karush–Kuhn–Tucker) conditions: for the lasso, every zero coefficient must have a gradient no larger than λ\lambda, and every non-zero one a gradient exactly equal to λ\lambda in size. They hold to within 10−610^{-6}, so the code really has found the minimum.

In practice λ\lambda is chosen by cross-validation (Resampling and simulation): fit the path on part of the data, measure prediction error on the rest, and pick the λ\lambda with the smallest average error, or, more conservatively, the largest λ\lambda within one standard error of the minimum. Because the lasso's shrinkage biases the kept coefficients, a common follow-up refits least squares on the selected variables (the relaxed lasso), at the price of more variance.

ComputationC

Principal component analysis

Penalties tame regression. Principal component analysis (PCA) tackles the other face of high dimensions: understanding the structure of the variables themselves. It finds the directions in which the data vary most. The first principal component is the unit vector v1v_1 that maximises the variance of the projected data Xv1X v_1; the second maximises it among directions orthogonal to the first; and so on. All of them come from one singular value decomposition of the centred data matrix, X=UDV⊤X = UDV^\top: the columns of VV are the loadings (directions), UDUD gives the scores (each sample's coordinates), and dj2/∑kdk2d_j^2/\sum_k d_k^2 is the share of variance explained by component jj. Karl Pearson described the idea in 1901 as finding "lines and planes of closest fit"; Harold Hotelling developed it as a statistical method in 1933.

In gene expression data, thousands of genes often move together because a few biological programmes, such as cell division or immune activity, switch many genes on and off at once. The simulation builds such data: 1,000 genes for 80 tumour samples of two subtypes, driven by two hidden programmes plus noise. It also runs PCA on pure noise of the same size, for comparison.

python
import numpy as np

rng = np.random.default_rng(1901)
n, p = 80, 1000                                         # 80 tumour samples, 1,000 genes
subtype = np.repeat([0, 1], 40)                         # two subtypes, 40 samples each
# Two hidden biological programmes drive many genes at once; everything else is noise.
f1 = rng.normal(0, 1, n) + 2.5 * subtype                # programme 1 differs between subtypes
f2 = rng.normal(0, 1, n)                                # programme 2 does not
load1 = np.where(np.arange(p) < 150, rng.normal(1.0, 0.3, p), 0.0)          # genes 0-149 follow f1
load2 = np.where((np.arange(p) >= 100) & (np.arange(p) < 300), rng.normal(0.8, 0.3, p), 0.0)
X = np.outer(f1, load1) + np.outer(f2, load2) + rng.normal(0, 1, (n, p))

def pca(X):
    Xc = X - X.mean(axis=0)
    U, d, Vt = np.linalg.svd(Xc, full_matrices=False)
    return U * d, Vt, d ** 2 / np.sum(d ** 2)           # scores, loadings, share of variance

scores, loadings, share = pca(X)
_, _, share_noise = pca(rng.normal(0, 1, (n, p)))
print("component:                 1      2      3      4      5      6")
print("share of variance, genes:", "".join(f"{s:7.1%}" for s in share[:6]))
print("share of variance, noise:", "".join(f"{s:7.1%}" for s in share_noise[:6]))
print(f"mean PC1 score by subtype: {scores[subtype == 0, 0].mean():6.1f} and {scores[subtype == 1, 0].mean():6.1f}")
print(f"mean PC2 score by subtype: {scores[subtype == 0, 1].mean():6.1f} and {scores[subtype == 1, 1].mean():6.1f}")
top = np.argsort(-np.abs(loadings[0]))[:20]
print("the 20 genes with the largest PC1 loadings are all among genes 0-149:", bool(np.all(top < 150)))

# A rough text plot of the samples on the first two components (A = subtype A, B = subtype B).
rows, cols = 11, 41
grid = [[" "] * cols for _ in range(rows)]
for (s1, s2), t in zip(scores[:, :2], subtype):
    c = int((s1 - scores[:, 0].min()) / np.ptp(scores[:, 0]) * (cols - 1))
    r = int((scores[:, 1].max() - s2) / np.ptp(scores[:, 1]) * (rows - 1))
    grid[r][c] = "AB"[t] if grid[r][c] in (" ", "AB"[t]) else "*"
print("PC2 ^\n" + "\n".join("    |" + "".join(row) for row in grid) + "\n    +" + "-" * cols + "> PC1")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The first two components explain 29.3% and 6.8% of the variance, and then the shares drop to about 1.3%: the two hidden programmes stand clearly above the noise. PC1 separates the subtypes (mean scores 14.5 and −14.5; the sign of a component is arbitrary), and its largest loadings point straight at the genes driven by the first programme. The noise matrix is the warning: even with no structure at all, its first component explains 2.0%, well above the 1.3% average, because with 1,000 dimensions and 80 samples some direction always happens to line up with the noise. In high dimensions, random-matrix theory (the Marchenko–Pastur law) describes how large noise components can be, and a component is worth interpreting only if it clearly exceeds that level.

PCA is unsupervised: it knows nothing about the outcome. Using the leading components as predictors (principal component regression) works when the outcome depends on the main directions of variation, and fails when the relevant signal lies in a direction of small variance. In genetics, the leading components of genotype data track ancestry, and genome-wide association studies include them as covariates so that differences in ancestry between cases and controls are not mistaken for genetic effects on disease.

Worked examplesH

False discoveries in genomics

The most common high-dimensional analysis is not a regression at all: it is the same simple test run thousands of times, once per gene, protein or genetic variant. You met the general problem in Multiple testing and reproducibility; genomics is where it reaches its full scale. The simulation compares two groups of 20 patients on 20,000 genes, 200 of which truly differ by 1.2 standard deviations.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(2)
genes, n, size = 20_000, 20, 1.2                        # 20 patients per group
effect = np.zeros(genes)
effect[:200] = rng.choice([-1, 1], 200) * size          # 200 genes truly differ, by 1.2 standard deviations
a = rng.normal(0, 1, (genes, n))
b = rng.normal(effect[:, None], 1, (genes, n))
t, pval = stats.ttest_ind(b, a, axis=1)
real = effect != 0

def report(name, called):
    false = np.sum(called & ~real)
    print(f"{name:28} {called.sum():6d} genes called   {np.sum(called & real):4d} real   {false:5d} false"
          f"   false discovery proportion {false / max(called.sum(), 1):.0%}")

report("p < 0.05", pval < 0.05)
report("Bonferroni, p < 0.05/20000", pval < 0.05 / genes)
# Benjamini-Hochberg at a false discovery rate of 10%.
order = np.argsort(pval)
passed = pval[order] <= 0.10 * np.arange(1, genes + 1) / genes
k = np.max(np.nonzero(passed)[0]) + 1 if passed.any() else 0
bh = np.zeros(genes, dtype=bool)
bh[order[:k]] = True
report("Benjamini-Hochberg, FDR 10%", bh)

# The winner's curse: effects estimated for the genes that were selected are too large.
diff = b.mean(axis=1) - a.mean(axis=1)
print(f"real genes selected by BH: average estimated |effect| {np.mean(np.abs(diff[bh & real])):.2f} (true {size})")
print(f"all 200 real genes:        average estimated |effect| {np.mean(np.abs(diff[real])):.2f} (true {size})")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

At p<0.05p \lt 0.05, 1,177 genes are called and 983 of them are false, close to the 19,800×0.05=99019{,}800 \times 0.05 = 990 expected: 84% of the "discoveries" are noise. Bonferroni's correction calls only 10 genes, all real, but misses 190 of the 200. The Benjamini–Hochberg procedure, controlling the false discovery rate at 10%, calls 128 genes, 114 real and 14 false; the false proportion of 11% is close to the 10% target, which is an average over repeated studies, not a guarantee for each one.

The last two lines show the winner's curse. The real genes that passed the threshold have an average estimated effect of 1.40 standard deviations, against a true 1.2. Across all 200 real genes the estimates average 1.21, essentially unbiased. Selection is the culprit: among real genes, those whose noise happened to push their estimate up were more likely to pass, so the selected estimates are inflated. Follow-up studies of "top hits" routinely find smaller effects for exactly this reason, and it is one reason genome-wide association studies require replication in an independent sample, and use a genome-wide significance threshold of 5×10−85 \times 10^{-8}, roughly a Bonferroni correction for the million or so independent common variants in the human genome. Results that pass are collected in the openly available GWAS Catalog.

Genomic data are among the most sensitive data about people: a genome identifies its owner and reveals information about relatives who never consented. Research access is controlled, consent governs use, and synthetic data like this lesson's are the right material for learning.

ExplanationH

Inference after selection

The winner's curse is one instance of a general problem. Classical confidence intervals and pp-values assume the hypothesis was fixed before seeing the data. In high-dimensional work we almost never do that: we run the lasso, see which variables it keeps, and then want intervals for their coefficients. Refitting least squares on the selected variables and reporting the textbook intervals uses the data twice, once to choose and once to test, and the intervals are too narrow and the pp-values too small, often dramatically. The field that repairs this is called selective inference or post-selection inference, and as of 2026 it is an active research area with several established approaches.

  • Sample splitting. Select on one half of the data, infer on the other. Simple and valid, at the cost of using only half the data for each task; repeating it over many random splits and combining the results recovers some of the loss.
  • Simultaneous inference. Richard Berk and colleagues' PoSI method (2013) widens intervals enough to be valid for every model that could have been selected. Always valid, often conservative.
  • Conditional inference. Compute the distribution of an estimate given that the selection procedure chose what it chose. For the lasso, Jason Lee, Jonathan Taylor and colleagues showed in 2016 that the selection event is a set of linear inequalities in yy, which yields exact conditional intervals.
  • Knockoffs. Rina Foygel Barber and Emmanuel Candès's knockoff filter (2015), and its "model-X" extension (2018), constructs fake copies of the predictors that mimic their correlations but are known to be unrelated to the outcome; a variable is selected only if it beats its own knockoff clearly, which controls the false discovery rate of the selected set.
  • Debiased estimators. The debiased (or desparsified) lasso corrects the lasso's shrinkage to give approximately normal estimates for individual coefficients, with valid confidence intervals under sparsity assumptions.

Related ideas protect predictions rather than coefficients: the frontier lesson Conformal prediction gives prediction intervals that remain valid whatever model or selection procedure produced the predictions.

History & global context

History and practice

Arthur Hoerl and Robert Kennard introduced ridge regression in 1970 for chemical-engineering data with nearly collinear predictors, long before "high-dimensional" was a phrase. Robert Tibshirani proposed the lasso in 1996; Bradley Efron and colleagues' least-angle regression (2004) showed how to compute its whole path, and coordinate descent made it routine. Signal processing arrived at the same mathematics from another direction: David Donoho, and Emmanuel Candès with Justin Romberg and Terence Tao, showed around 2006 that sparse signals can be recovered from far fewer measurements than classical sampling requires (compressed sensing), which now shortens MRI scans. Yoav Benjamini and Yosef Hochberg's 1995 false discovery rate, at first little used, became indispensable with the arrival of microarrays a few years later.

Today the standard tools are free and well documented: glmnet in R, scikit-learn's linear models and PCA in Python, and equivalents in Julia and other languages. Good practice is the same everywhere: standardise predictors, choose penalties by cross-validation that repeats the entire pipeline (including any screening of variables) inside each fold, report how stable the selected set is across resamples (Nicolai Meinshausen and Peter Bühlmann's stability selection formalises this), keep a held-out test set for the final estimate of accuracy, and do not report naive pp-values for selected variables.

Common misconceptionsH

Common misconceptions

Misconception

"The lasso finds the true variables." It finds a sparse set that predicts well. Recovering exactly the true set requires strong conditions: real effects large enough to stand out from the noise, and true and irrelevant predictors not too correlated (the "irrepresentable condition"). With correlated genes, the selected set can change from one sample to the next while predictions barely change.

Misconception

"PCA finds the variables that matter." PCA finds directions of large variance in the predictors, without looking at the outcome. The leading components may be dominated by batch effects (which laboratory processed the sample, on which day) rather than by biology, and the signal that predicts the outcome may lie in a direction of small variance.

Misconception

"Screening first, then cross-validating, is fine." If you select the 100 genes most correlated with the outcome using all the data, and only then cross-validate a model on those 100, the test folds have already influenced the selection, and the estimated error is optimistic, sometimes wildly so. The screening must be repeated inside every fold.

Misconception

"A penalised model's coefficients can be read like ordinary regression coefficients." They are deliberately biased towards zero, their size depends on λ\lambda and on the scaling convention, and their standard errors from ordinary formulas are invalid. Interpret them as parts of a prediction rule, and use the methods of selective inference for formal claims about individual variables.

Exercises

Exercises

1. High-dimensional statistics

Warm-upHuman

A team in Seoul has gene-expression measurements on 5,000 genes for 60 patients and fits an ordinary least-squares regression of a blood marker on all 5,000 genes. What happens?

Choose one

2. High-dimensional statistics

CoreHuman

With an orthonormal design (X⊤X/n=IX^\top X/n = I), the least-squares estimates of four coefficients are 3.03.0, −0.4-0.4, 1.11.1 and −2.5-2.5. The lasso with λ=1\lambda = 1 (in the convention 12n∥y−Xβ∥2+λ∥β∥1\frac{1}{2n}\|y - X\beta\|^2 + \lambda\|\beta\|_1) is fitted. How many of its four coefficients are non-zero?

3. High-dimensional statistics

Warm-upHuman

A genome-wide scan in Nairobi tests 500,000 genetic variants for association with a trait. Suppose that, in truth, none of them is associated. Using the threshold p<10−4p \lt 10^{-4} for each test, how many variants would you expect to be declared significant?

4. High-dimensional statistics

CoreHuman

Researchers fit the lasso to 2,000 candidate predictors of hospital readmission, keep the 12 it selects, refit ordinary least squares on those 12 using the same data, and report the usual 95% confidence intervals and pp-values. What is the main problem?

Choose one

5. High-dimensional statistics

StretchHuman

With 50 patients and 10,000 genes of pure noise (unrelated to the outcome), an analyst picks the 20 genes most correlated with the outcome using all 50 patients, then estimates the accuracy of a classifier built on those 20 genes by 5-fold cross-validation. What will the cross-validated accuracy look like?

Choose one

6. High-dimensional statistics

CoreComputational

Implement the lasso by coordinate descent.

  1. soft_threshold(z, t) returns sign(z)max⁡(∣z∣−t,0)\text{sign}(z)\max(|z| - t, 0) as a float.
  2. lasso_cd(X, y, lam) minimises 12n∥y−Xβ∥2+λ∥β∥1\frac{1}{2n}\|y - X\beta\|^2 + \lambda\|\beta\|_1, assuming the columns of X are standardised so that xj⊤xj/n=1x_j^\top x_j/n = 1 (and y is centred, so there is no intercept). Start from β=0\beta = 0 and repeat sweeps over j=1,…,pj = 1, \ldots, p, setting βj←S(xj⊤r/n+βj, λ)\beta_j \leftarrow S(x_j^\top r/n + \beta_j,\ \lambda) where rr is the current residual, until no coefficient changes by more than 10−810^{-8}. Return the coefficients as a NumPy array.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. High-dimensional statistics

CoreComputational

Write pca_variance_share(X), which takes an n×pn \times p data matrix, centres each column, and returns a NumPy array of the shares of total variance explained by the principal components, in decreasing order (they should add up to 1). Use the singular value decomposition: the share of component jj is dj2/∑kdk2d_j^2 / \sum_k d_k^2.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources