Skip to content
LumenaryAcademy
Modelling 80 min

Likelihood and model selection

The likelihood function as the common thread of statistical models — maximum likelihood by numerical optimisation, likelihood-ratio tests, AIC, BIC and cross-validation, overfitting seen in action, and what parsimony can and cannot promise.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

One principle behind every fit

The previous lessons fitted models by three apparently different recipes: least squares for linear regression, Newton's method on a log-likelihood for logistic and Poisson regression, and sums of squares for analysis of variance. They are one idea in three costumes. The idea is the likelihood: a model assigns a probability to the data we observed, and parameter values under which our data would have been more probable are better supported by them.

Likelihood is the most widely used tool of statistical modelling. It gives estimates for models with no formula for the answer (the shape of a distribution of wind speeds, the parameters of an epidemic curve), standard errors for those estimates, tests that compare models, and the starting point of Bayesian inference. It also raises the second question of this lesson. Given several models, simple and complicated, all fitted to the same data, which should we believe? A more complicated model always fits the data in hand better. Does it describe the world better? Answering that honestly, with AIC, BIC and cross-validation, is the subject of model selection, and it is where statistics meets machine learning.

DefinitionsH

The likelihood function

Definition— likelihood

Let f(y;θ)f(\mathbf y; \theta) be the probability (or density) of data y\mathbf y under a model with parameter θ\theta. Once y\mathbf y is observed, the likelihood function is L(θ)=f(y;θ)L(\theta) = f(\mathbf y; \theta), viewed as a function of θ\theta with the data fixed. For independent observations it is a product, L(θ)=∏if(yi;θ)L(\theta) = \prod_i f(y_i; \theta), so the log-likelihood ℓ(θ)=log⁡L(θ)=∑ilog⁡f(yi;θ)\ell(\theta) = \log L(\theta) = \sum_i \log f(y_i; \theta) is a sum. Constant factors that do not depend on θ\theta can be dropped.

Definition— maximum-likelihood estimate

The maximum-likelihood estimate (MLE) θ^\hat\theta is the value of θ\theta that maximises L(θ)L(\theta), equivalently ℓ(θ)\ell(\theta). The score is ℓ′(θ)\ell'(\theta), and the observed information J(θ^)=−ℓ′′(θ^)J(\hat\theta) = -\ell''(\hat\theta) measures how sharply the log-likelihood curves at its peak.

A first example. Thandiwe tests a new batch of bean seeds: 7 of 10 germinate. If seeds germinate independently with probability pp, the probability of this particular outcome is p7(1−p)3p^7(1-p)^3 (times a constant counting the orders, which we drop), so ℓ(p)=7log⁡p+3log⁡(1−p)\ell(p) = 7\log p + 3\log(1-p). Setting ℓ′(p)=7/p−3/(1−p)=0\ell'(p) = 7/p - 3/(1-p) = 0 gives p^=0.7\hat p = 0.7: the MLE is the observed proportion, as intuition says. Likelihood is not a probability distribution over pp: it need not integrate to 1, and only ratios of likelihoods mean anything. Comparing p=0.5p = 0.5 with p=0.7p = 0.7, the data are 0.4390.439 times as probable under the first, which is weak evidence against p=0.5p = 0.5.

python
import numpy as np
from scipy import stats, optimize

# 7 of 10 seeds germinated. Log-likelihood of the germination probability p (constants dropped):
def loglik(p, k, n):
    return k * np.log(p) + (n - k) * np.log(1 - p)

for p in [0.3, 0.5, 0.6, 0.7, 0.8, 0.9]:
    print(f"p = {p:.1f}: likelihood {np.exp(loglik(p, 7, 10)):.5f}, "
          f"relative to the maximum {np.exp(loglik(p, 7, 10) - loglik(0.7, 7, 10)):.3f}")

def lr_interval(k, n, level=0.95):
    """All p whose log-likelihood is within chi2(1) quantile / 2 of the maximum."""
    p_hat, cut = k / n, stats.chi2.ppf(level, 1)
    g = lambda p: 2 * (loglik(p_hat, k, n) - loglik(p, k, n)) - cut
    return optimize.brentq(g, 1e-12, p_hat), optimize.brentq(g, p_hat, 1 - 1e-12)

for k, n in [(7, 10), (70, 100)]:
    p_hat = k / n
    se = np.sqrt(p_hat * (1 - p_hat) / n)
    lo, hi = lr_interval(k, n)
    print(f"{k:2d} of {n:3d}: likelihood-ratio 95% interval {lo:.3f} to {hi:.3f};"
          f" Wald interval {p_hat - 1.96 * se:.3f} to {p_hat + 1.96 * se:.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Two features of maximum likelihood make it the default estimator, under regularity conditions (a smooth log-likelihood, a parameter inside its range rather than on a boundary, a number of parameters that does not grow with nn).

Theorem— Large-sample behaviour of the MLE

As n→∞n \to \infty, θ^\hat\theta is consistent (it converges to the true θ\theta) and asymptotically normal: approximately θ^∼N(θ,J(θ^)−1)\hat\theta \sim N\big(\theta, J(\hat\theta)^{-1}\big), so SE(θ^)≈1/J(θ^)\text{SE}(\hat\theta) \approx 1/\sqrt{J(\hat\theta)}, with the inverse information matrix in several dimensions. Its asymptotic variance attains the Cramér–Rao lower bound, the smallest possible for unbiased estimators under the same conditions. And the MLE is invariant: the MLE of g(θ)g(\theta) is g(θ^)g(\hat\theta).

For the seeds, J(p^)=n/(p^(1−p^))J(\hat p) = n/(\hat p(1-\hat p)), so SE=p^(1−p^)/n\text{SE} = \sqrt{\hat p(1 - \hat p)/n}, the familiar formula behind the Wald interval p^±1.96 SE\hat p \pm 1.96\,\text{SE}. The program also computes a likelihood-ratio interval: all values of pp whose log-likelihood is within 1.921.92 of the maximum (half of 3.84, the 95% point of χ12\chi^2_1, for reasons the next section explains). With 7 of 10 seeds the Wald interval, 0.416 to 0.984, is symmetric and reaches almost to 1; the likelihood-ratio interval, 0.393 to 0.915, follows the asymmetric shape of the likelihood. With 70 of 100, the two nearly agree (0.606–0.784 and 0.610–0.790), because the log-likelihood has become nearly quadratic, which is exactly what asymptotic normality means.

0.00.20.40.60.81.000.51relative likelihood 0.147MLE 0.7germination probability pL(p) / L(0.7)7 of 10 seeds70 of 100 seeds
Relative likelihood of the germination probability: 7 of 10 seeds (solid) and 70 of 100 (dashed). Both peak at the maximum-likelihood estimate 0.7, but ten times the data give a much narrower curve. The shaded band is the 95% likelihood-ratio interval from 7 of 10 seeds, all values whose likelihood is at least 0.147 of the maximum.

For the normal linear model, maximising the likelihood over β\boldsymbol\beta means minimising ∑(yi−xi⊤β)2\sum (y_i - \mathbf x_i^\top\boldsymbol\beta)^2: least squares is maximum likelihood when errors are normal, and the logistic and Poisson fits of the previous lesson were maximum likelihood with other distributions. The deviance used there is −2-2 times a log-likelihood ratio.

ComputationC

Maximum likelihood by numerical optimisation

Most likelihoods have no closed-form maximum, and we maximise them numerically. A classic case comes from wind energy. Engineers assessing a site in, say, the Atacama or on the coast of Tamil Nadu summarise a year of hourly wind speeds by a Weibull distribution, with density f(v)=kc(vc)k−1e−(v/c)kf(v) = \frac{k}{c}\left(\frac{v}{c}\right)^{k-1} e^{-(v/c)^k} for v>0v > 0: a scale cc near the typical speed and a shape kk describing how variable the wind is. The fitted distribution feeds the estimate of energy output, which depends heavily on the frequency of strong winds.

The program fits a Weibull distribution to 300 synthetic hourly speeds by minimising the negative log-likelihood with SciPy's BFGS optimiser. Two practical habits are visible. The parameters are optimised on the log scale, so the optimiser cannot wander into negative shapes or scales. And the answer is checked against an independent implementation, scipy.stats.weibull_min.fit. Standard errors come from the observed information, the matrix of second derivatives of the negative log-likelihood at the optimum, computed by finite differences. Finally, the program tests two simpler models nested inside the Weibull: the exponential distribution (k=1k = 1) and the Rayleigh distribution (k=2k = 2), which wind engineers often assume when data are scarce.

python
import numpy as np
from scipy import stats, optimize

# 300 synthetic hourly mean wind speeds (m/s) at a candidate wind-farm site.
rng = np.random.default_rng(2024)
v = stats.weibull_min.rvs(2.1, scale=7.5, size=300, random_state=rng)

def nll(theta):
    """Negative log-likelihood of the Weibull(shape k, scale c), with theta = (log k, log c) so both stay positive."""
    k, c = np.exp(theta)
    return -np.sum(np.log(k / c) + (k - 1) * np.log(v / c) - (v / c) ** k)

res = optimize.minimize(nll, x0=[0.0, np.log(v.mean())], method="BFGS")
k_hat, c_hat = np.exp(res.x)
print(f"our MLE:    shape k = {k_hat:.3f}, scale c = {c_hat:.3f} m/s  (log-likelihood {-res.fun:.3f})")
k_sp, _, c_sp = stats.weibull_min.fit(v, floc=0)
print(f"SciPy fit:  shape k = {k_sp:.3f}, scale c = {c_sp:.3f} m/s")

# Standard errors from the observed information: the Hessian of nll in (k, c), by central differences.
def nll_kc(p):
    return nll(np.log(p))
p0, h = np.array([k_hat, c_hat]), 1e-4
H = np.zeros((2, 2))
for i in range(2):
    for j in range(2):
        ei, ej = np.eye(2)[i] * h, np.eye(2)[j] * h
        H[i, j] = (nll_kc(p0 + ei + ej) - nll_kc(p0 + ei - ej) - nll_kc(p0 - ei + ej) + nll_kc(p0 - ei - ej)) / (4 * h * h)
se = np.sqrt(np.diag(np.linalg.inv(H)))
print(f"standard errors: k {se[0]:.3f}, c {se[1]:.3f};  95% CI for k: {k_hat - 1.96 * se[0]:.2f} to {k_hat + 1.96 * se[0]:.2f}")

# Likelihood-ratio tests of two simpler models nested in the Weibull
ll_full = -res.fun
for name, k0 in [("exponential (k = 1)", 1.0), ("Rayleigh (k = 2)", 2.0)]:
    fixed = optimize.minimize_scalar(lambda logc: nll([np.log(k0), logc]), bounds=(-5, 5), method="bounded")
    lr = 2 * (ll_full + fixed.fun)
    print(f"{name:>20}: LR = {lr:7.2f} on 1 df, p = {stats.chi2.sf(lr, 1):.2g}")
print(f"fitted P(v > 3 m/s, a typical cut-in speed) = {np.exp(-(3 / c_hat) ** k_hat):.3f}; observed {np.mean(v > 3):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Our optimiser and SciPy agree: k^=2.080\hat k = 2.080 and c^=7.195\hat c = 7.195 m/s, with standard errors 0.093 and 0.211. The exponential model is rejected overwhelmingly (likelihood ratio 188.10), while the Rayleigh model fits about as well as the full Weibull (p=0.39p = 0.39): with these data, the simpler model is good enough. The fitted distribution says the wind exceeds 3 m/s, a typical turbine cut-in speed, 85.0% of the time, against 86.0% in the data.

Numerical maximum likelihood has failure modes worth knowing: an optimiser may stop at a local maximum (try several starting points), the likelihood may be flat in some direction because two parameters trade off against each other (the model is barely identifiable, and standard errors explode), or the maximum may sit on the boundary of the parameter space. The theory of optimisation in Lumenary Math explains when such methods converge.

ComputationC

Likelihood-ratio tests and Wilks' theorem

Comparing a model with a simpler one nested inside it (obtained by fixing some parameters) is the job of the likelihood-ratio test. The bigger model always has the higher maximised likelihood; the question is whether the gain is more than chance would produce.

Theorem— Wilks' theorem

Suppose model M0M_0, with p0p_0 free parameters, is nested in model M1M_1, with p1p_1, and M0M_0 is true. Under regularity conditions, as n→∞n \to \infty,

Λ=2(ℓ1(θ^1)−ℓ0(θ^0))  ⟶  χp1−p02\Lambda = 2\big(\ell_1(\hat\theta_1) - \ell_0(\hat\theta_0)\big) \;\longrightarrow\; \chi^2_{p_1 - p_0}

in distribution.

The idea of the proof is a Taylor expansion: near its maximum the log-likelihood is approximately quadratic, the MLE approximately normal, and twice the drop in a quadratic from its maximum, along p1−p0p_1 - p_0 directions, is a sum of p1−p0p_1 - p_0 squared standard normal variables. Samuel Wilks proved it in 1938. The theorem also justifies the likelihood-ratio interval above (the values of θ\theta that such a test would not reject), and the deviance tests of the previous lesson. Is the approximation good at realistic sample sizes? The program simulates 1000 samples of 50 wind speeds from a Rayleigh distribution, so that the simpler model is true, and compares the likelihood-ratio statistic with χ12\chi^2_1.

python
import numpy as np
from scipy import stats, optimize

# Wilks' theorem by simulation: when the simpler model is true, is 2 * (gain in log-likelihood) ~ chi-squared(1)?
rng = np.random.default_rng(1938)

def weibull_loglik(k, c, v):
    return np.sum(np.log(k / c) + (k - 1) * np.log(v / c) - (v / c) ** k)

def lr_statistic(v):
    full = optimize.minimize(lambda t: -weibull_loglik(*np.exp(t), v), x0=[np.log(2.0), np.log(v.mean())], method="BFGS")
    c_rayleigh = np.sqrt(np.mean(v ** 2))            # the MLE of the scale when the shape is fixed at 2
    return 2 * (-full.fun - weibull_loglik(2.0, c_rayleigh, v))

lr = np.array([lr_statistic(stats.weibull_min.rvs(2.0, scale=7.5, size=50, random_state=rng)) for _ in range(1000)])
print(f"{'quantile':>8} {'simulated':>10} {'chi-squared(1)':>15}")
for q in [0.5, 0.9, 0.95, 0.99]:
    print(f"{q:8.2f} {np.quantile(lr, q):10.2f} {stats.chi2.ppf(q, 1):15.2f}")
print(f"share of LR > 3.84 (a 5% test): {np.mean(lr > stats.chi2.ppf(0.95, 1)):.1%}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
ExplanationH

AIC and BIC

Likelihood-ratio tests compare two nested models. Often we have many candidate models, not all nested, and the goal is not to test a hypothesis but to choose a model that will serve well. The maximised log-likelihood cannot choose on its own, because it always favours the biggest model: each extra parameter can only raise it.

Hirotugu Akaike, at the Institute of Statistical Mathematics in Tokyo, asked in the early 1970s how well a fitted model would predict new data from the same process, measured by the expected log-likelihood of new data, which is related to the Kullback–Leibler divergence from the truth (see information theory). The maximised log-likelihood is an optimistic estimate of that, because the parameters were tuned to the data in hand, and he showed that the optimism is approximately kk, the number of estimated parameters. Correcting for it, and multiplying by −2-2 by convention, gives

AIC=−2ℓ(θ^)+2k,BIC=−2ℓ(θ^)+klog⁡n.\text{AIC} = -2\ell(\hat\theta) + 2k, \qquad \text{BIC} = -2\ell(\hat\theta) + k\log n.

The Bayesian information criterion, proposed by Gideon Schwarz in 1978, comes from a different question: which model has the highest posterior probability, with equal prior probabilities on the models? For large nn, −12BIC-\tfrac12\text{BIC} approximates the log of the marginal likelihood of the data under the model (Bayesian inference explains marginal likelihoods). Smaller is better for both, and only differences matter: a difference in AIC of about 2 or less is weak evidence, while 10 or more is strong.

The two criteria want different things. BIC's penalty, log⁡n\log n per parameter, exceeds AIC's 2 once n≥8n \geq 8, so BIC chooses simpler models. If one candidate is the true model, BIC finds it with probability tending to 1 (it is consistent), while AIC keeps a positive chance of choosing a model that is too big. But the true model is rarely among the candidates. When the goal is prediction and reality is more complex than any candidate, AIC's willingness to add parameters as data accumulate is an advantage, and it is asymptotically efficient for prediction. Neither is "right"; they answer different questions. For small samples, the corrected AICc=AIC+2k(k+1)n−k−1\text{AIC}_c = \text{AIC} + \frac{2k(k+1)}{n - k - 1} is preferable.

Both criteria compare models of the same data: the same observations, the same response on the same scale. Comparing the AIC of a model for rent with that of a model for log rent is meaningless without a correction for the transformation.

ComputationC

Cross-validation and overfitting

The most direct way to estimate how a model will do on new data is to hold some data back. In KK-fold cross-validation the data are split at random into KK parts (folds); each part in turn is left out, the model is fitted to the other K−1K - 1, and its prediction error is measured on the left-out part. The average over folds estimates the out-of-sample error. Leave-one-out cross-validation (LOO-CV) is the case K=nK = n. For linear least squares it needs no refitting at all: the residual that observation ii would have if it were left out is ei/(1−hii)e_i/(1 - h_{ii}), with hiih_{ii} the leverage from the hat matrix, so

LOO-CV=1n∑i=1n(ei1−hii)2.\text{LOO-CV} = \frac{1}{n}\sum_{i=1}^n \left(\frac{e_i}{1 - h_{ii}}\right)^2.

Cross-validation assumes almost nothing about the model (it works for neural networks and random forests as well as for regressions), and Mervyn Stone showed in 1977 that LOO-CV and AIC are asymptotically equivalent for comparing models. The price is computation, and some care, discussed below, about what counts as "new" data.

Here is overfitting in action. Twenty noisy points come from a smooth curve, y=sin⁡3x+εy = \sin 3x + \varepsilon with noise standard deviation 0.3, and polynomials of degree 0 to 10 are fitted to them. Because the data are synthetic, we can also measure each fitted curve's true error on 5000 fresh points.

python
import numpy as np

# 20 noisy observations of a smooth curve; fit polynomials of degree 0 to 10.
rng = np.random.default_rng(1974)
f = lambda x: np.sin(3 * x)
n, sigma = 20, 0.3
x = rng.uniform(-1, 1, n)
y = f(x) + rng.normal(0, sigma, n)
x_new = rng.uniform(-1, 1, 5000)                       # fresh data from the same process, for the "truth"
y_new = f(x_new) + rng.normal(0, sigma, 5000)
folds = rng.permutation(n) % 5                          # 5-fold cross-validation, same folds for every degree

rows = []
for d in range(11):
    X = np.vander(x, d + 1)
    b = np.linalg.lstsq(X, y, rcond=None)[0]
    sse = np.sum((y - X @ b) ** 2)
    k = d + 2                                           # d + 1 coefficients plus the error variance
    loglik = -n / 2 * (np.log(2 * np.pi * sse / n) + 1)  # maximised Gaussian log-likelihood
    h = np.diag(X @ np.linalg.pinv(X))                  # leverages, for the leave-one-out shortcut
    loo = np.mean(((y - X @ b) / (1 - h)) ** 2)
    cv5 = 0.0
    for fold in range(5):
        tr, te = folds != fold, folds == fold
        bf = np.linalg.lstsq(X[tr], y[tr], rcond=None)[0]
        cv5 += np.sum((y[te] - X[te] @ bf) ** 2) / n
    test = np.mean((y_new - np.vander(x_new, d + 1) @ b) ** 2)
    rows.append((d, sse / n, test, -2 * loglik + 2 * k, -2 * loglik + k * np.log(n), loo, cv5))

print(f"{'degree':>6} {'train MSE':>9} {'test MSE':>9} {'AIC':>6} {'BIC':>6} {'LOO-CV':>8} {'5-fold':>8}")
for r in rows:
    print(f"{r[0]:6d} {r[1]:9.4f} {r[2]:9.4f} {r[3]:6.1f} {r[4]:6.1f} {r[5]:8.4g} {r[6]:8.4g}")
R = np.array(rows)
for j, name in [(1, "training error"), (2, "test error"), (3, "AIC"), (4, "BIC"), (5, "LOO-CV"), (6, "5-fold CV")]:
    print(f"{name:>14} picks degree {int(R[np.argmin(R[:, j]), 0])}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
0123456789100.050.10.20.51noise variance 0.09: no model can do better on new databest on new data: degree 3error on the 20 fitted pointserror on 5000 new pointsdegree of the polynomialmean squared error (log scale)
Error on the 20 points used for fitting (solid) and on 5000 new points from the same process (dashed), for polynomials of degree 0 to 10, on a log scale. Training error can only fall as the degree rises. Error on new data falls until degree 3, stays nearly flat, then rises steeply as the polynomial starts to fit the noise; at degree 10 it is nine times its best value. The dotted line is the noise variance, 0.09, below which no model can predict new data.

The training error falls with every added degree, from 0.557 to 0.056: by that measure the degree-10 polynomial is the best model, and it is the worst. Its error on new data is 0.950, nine times the best achievable here. The true error is smallest at degree 3 (0.102, close to the noise floor of 0.09), and AIC, BIC, leave-one-out and 5-fold cross-validation all choose degree 3, without ever seeing the 5000 new points. Notice also how the cross-validation errors explode for high degrees (to 347.5 for 5-fold at degree 10): leaving out a point near the edge of the data forces a high-degree polynomial to extrapolate, and it swings wildly. That is overfitting made visible.

Two rules keep cross-validation honest. Everything that learns from the data must happen inside the folds: choosing variables, scaling, imputing missing values or tuning settings on the whole dataset, then cross-validating, leaks information from the held-out folds and makes the estimate optimistic. And the folds must respect the structure of the data: with repeated measurements of the same patients, leave out whole patients; with time series, never train on the future to predict the past. If a model's settings are tuned by cross-validation, an honest estimate of its performance needs a second, outer layer (nested cross-validation) or a test set locked away until the end.

ExplanationH

Parsimony and its limits

Every method in this lesson rewards parsimony: a model should be no more complicated than the data can support. This is a statistical version of Occam's razor, and it has a precise justification, the bias–variance trade-off. A simple model may be biased, missing real structure; a complex one has low bias but high variance, because its many parameters are each estimated from limited information and chase the noise. The best predictions balance the two, and how much complexity the data can support grows with the amount of data.

George Box put the attitude in a famous phrase in 1979: all models are wrong, but some are useful. Model selection does not find the true model; it finds a useful one for a stated purpose. Several limits follow, and they are where good statisticians earn their keep.

  • Selection invalidates naive inference. If you try many models and report the standard errors and pp-values of the one you chose, those numbers are too optimistic, because they ignore the selection. This is one of Andrew Gelman and Eric Loken's "forking paths": even without deliberate fishing, analysis choices that depend on the data make pp-values unreliable. The remedies are to pre-register the analysis, to split the data (select on one part, infer on the other), to average over models, or to use methods of post-selection inference designed for the purpose. See multiple testing and reproducibility.
  • The best predictive model is not the best explanatory one. A model chosen by AIC or cross-validation to predict well may drop a confounder that matters for causal interpretation, or keep a proxy that predicts without explaining. Selection criteria answer "what predicts?", not "what causes?" (causal inference).
  • Sometimes there is no single best model. When several models are within a couple of AIC units of each other, choosing one and discarding the rest understates uncertainty. Model averaging, weighting the predictions of several models, often predicts better and is more honest.
  • Shrinkage is often better than selection. Rather than including or excluding each parameter, regularisation (ridge regression, the lasso) keeps many parameters but shrinks them towards zero, choosing the amount of shrinkage by cross-validation; this is the main tool when there are many predictors, in high-dimensional statistics.
  • Modern machine learning stretches the classical picture. Deep neural networks often have far more parameters than data points, fit the training data exactly, and still predict well, a phenomenon studied under names such as "benign overfitting" and "double descent". The bias–variance trade-off is not repealed, but parameter count turns out to be a poor measure of complexity for such models. Statistical learning theory takes this up.
ExplanationH

The professional view

In R, logLik(), AIC() and BIC() work for almost every fitted model, anova(fit0, fit1, test = "LRT") performs likelihood-ratio tests between nested GLMs, and optim() maximises custom likelihoods. In Python, statsmodels results carry .llf, .aic and .bic, SciPy's optimize.minimize handles custom likelihoods, and scikit-learn provides cross-validation splitters, including grouped and time-series splits, and tools for nested cross-validation. In Bayesian work with Stan and similar tools, approximate leave-one-out cross-validation by Pareto-smoothed importance sampling, implemented in the loo package, has largely replaced older information criteria; that is the subject of the Markov chain Monte Carlo lesson.

In applied work, the most important decisions come before any criterion is computed: which candidate models are scientifically plausible, what the model is for (prediction, explanation, a decision), and how its performance will be judged on data it has never seen. In regulated fields such as medical devices, credit scoring and official statistics, the validation plan is usually written down before the model is built, and the final test data are kept away from the modellers.

Common misconceptionsH

Common misconceptions

Misconception

"The likelihood is the probability that the parameter is correct." L(θ)L(\theta) is the probability of the data under θ\theta, read as a function of θ\theta. It does not integrate to 1 over θ\theta, and turning it into probabilities about θ\theta requires a prior distribution, which is Bayesian inference.

Misconception

"The model with the lowest AIC is the true model." AIC estimates relative predictive accuracy among the candidates you supplied. All of them may be wrong, and models within about 2 units are practically indistinguishable.

Misconception

"Cross-validation error measures how the final model will do in deployment." Only if the folds mimic deployment. Random folds on time-ordered data, or on repeated measurements of the same people, or after selecting features on all the data, give optimistic estimates. And a model chosen as the best of many by cross-validation is itself optimistically evaluated by that same cross-validation.

Misconception

"Overfitting means the model is too big." Overfitting means fitting noise, and it depends on the model's flexibility relative to the information in the data. Twenty points cannot support a degree-10 polynomial; twenty thousand could. And a large model with suitable regularisation can generalise well.

Exercises

Exercises

1. Likelihood and model selection

Warm-upHuman

In the lesson, 7 of 10 seeds germinated, and L(0.5)/L(0.7)=0.439L(0.5)/L(0.7) = 0.439. Which statement is correct?

Choose one

2. Likelihood and model selection

Warm-upHuman

A clinic in Kampala records the number of patients arriving with malaria on five Mondays: 3, 5, 4, 0, 8. Modelling the counts as independent Poisson(λ\lambda), what is the maximum-likelihood estimate of λ\lambda?

3. Likelihood and model selection

CoreHuman

Two models are fitted to the same 100 observations. Model A has 3 parameters and maximised log-likelihood −120-120; model B has 5 parameters and maximised log-likelihood −118-118. What is the AIC of model B? (Then compare it with A's.)

4. Likelihood and model selection

CoreHuman

A Poisson regression of hospital admissions has 4 coefficients; adding 3 air-pollution measures gives a model with 7. The maximised log-likelihoods are −210.4-210.4 and −204.1-204.1. What does a likelihood-ratio test at the 5% level conclude? (The 95% point of χ32\chi^2_3 is 7.81.)

Choose one

5. Likelihood and model selection

CoreHuman

An analyst has 80 patients and 5000 gene-expression measurements. She first picks the 20 genes most correlated with the outcome using all 80 patients, then fits a model on those 20 genes and estimates its accuracy by 10-fold cross-validation, getting 90%. What is the main problem?

Choose one

6. Likelihood and model selection

StretchHuman

Which statements about AIC and BIC are correct? Select all that apply.

Choose all that apply

7. Likelihood and model selection

CoreComputational

Rainfall amounts on wet days are often modelled by a gamma distribution with shape aa and scale ss, density f(x)=xa−1e−x/sΓ(a) saf(x) = \dfrac{x^{a-1}e^{-x/s}}{\Gamma(a)\,s^a} for x>0x > 0. Write fit_gamma(x) that returns the maximum-likelihood estimates (a, s) by minimising the negative log-likelihood with scipy.optimize.minimize. Optimise over (log⁡a,log⁡s)(\log a, \log s) so that both stay positive, and use scipy.special.gammaln for log⁡Γ(a)\log\Gamma(a). The tests compare your answer with scipy.stats.gamma.fit(x, floc=0).

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Likelihood and model selection

StretchComputational

Write loocv_mse(X, y) that returns the leave-one-out cross-validation mean squared error of a least-squares fit without refitting nn times, using the shortcut from the lesson: 1n∑i(ei/(1−hii))2\frac1n\sum_i \big(e_i/(1 - h_{ii})\big)^2, where eie_i are the ordinary residuals and hiih_{ii} the leverages (the diagonal of H=X(X⊤X)−1X⊤H = X(X^\top X)^{-1}X^\top). The tests compare your answer with brute-force refits.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources

  • SciPy: optimisation (scipy.optimize)The minimisers and root finders used in this lesson.
  • SciPy: statistical functions (scipy.stats)Every distribution has a fit method for maximum likelihood.
  • scikit-learn: cross-validationk-fold, grouped and time-series cross-validation, and nested cross-validation for tuned models.
  • A new look at the statistical model identification · Hirotugu Akaike · 1974IEEE Transactions on Automatic Control, volume 19. The paper that introduced AIC to a wide audience.
  • Estimating the dimension of a model · Gideon Schwarz · 1978Annals of Statistics, volume 6. The Bayesian information criterion.
  • The Elements of Statistical Learning · Trevor Hastie, Robert Tibshirani and Jerome FriedmanFree online. Its chapter on model assessment and selection covers AIC, BIC, cross-validation and the bias–variance trade-off in depth.
  • loo: efficient leave-one-out cross-validation for Bayesian modelsThe Stan project's package for approximate leave-one-out cross-validation, with its documentation and papers.
  • The garden of forking paths · Andrew Gelman and Eric Loken · 2013Why data-dependent choices of analysis invalidate p-values, even without any deliberate fishing.