One principle behind every fit
The previous lessons fitted models by three apparently different recipes: least squares for linear regression, Newton's method on a log-likelihood for logistic and Poisson regression, and sums of squares for analysis of variance. They are one idea in three costumes. The idea is the likelihood: a model assigns a probability to the data we observed, and parameter values under which our data would have been more probable are better supported by them.
Likelihood is the most widely used tool of statistical modelling. It gives estimates for models with no formula for the answer (the shape of a distribution of wind speeds, the parameters of an epidemic curve), standard errors for those estimates, tests that compare models, and the starting point of Bayesian inference. It also raises the second question of this lesson. Given several models, simple and complicated, all fitted to the same data, which should we believe? A more complicated model always fits the data in hand better. Does it describe the world better? Answering that honestly, with AIC, BIC and cross-validation, is the subject of model selection, and it is where statistics meets machine learning.
The likelihood function
Definition— likelihood
Let be the probability (or density) of data under a model with parameter . Once is observed, the likelihood function is , viewed as a function of with the data fixed. For independent observations it is a product, , so the log-likelihood is a sum. Constant factors that do not depend on can be dropped.
Definition— maximum-likelihood estimate
The maximum-likelihood estimate (MLE) is the value of that maximises , equivalently . The score is , and the observed information measures how sharply the log-likelihood curves at its peak.
A first example. Thandiwe tests a new batch of bean seeds: 7 of 10 germinate. If seeds germinate independently with probability , the probability of this particular outcome is (times a constant counting the orders, which we drop), so . Setting gives : the MLE is the observed proportion, as intuition says. Likelihood is not a probability distribution over : it need not integrate to 1, and only ratios of likelihoods mean anything. Comparing with , the data are times as probable under the first, which is weak evidence against .
import numpy as np
from scipy import stats, optimize
# 7 of 10 seeds germinated. Log-likelihood of the germination probability p (constants dropped):
def loglik(p, k, n):
return k * np.log(p) + (n - k) * np.log(1 - p)
for p in [0.3, 0.5, 0.6, 0.7, 0.8, 0.9]:
print(f"p = {p:.1f}: likelihood {np.exp(loglik(p, 7, 10)):.5f}, "
f"relative to the maximum {np.exp(loglik(p, 7, 10) - loglik(0.7, 7, 10)):.3f}")
def lr_interval(k, n, level=0.95):
"""All p whose log-likelihood is within chi2(1) quantile / 2 of the maximum."""
p_hat, cut = k / n, stats.chi2.ppf(level, 1)
g = lambda p: 2 * (loglik(p_hat, k, n) - loglik(p, k, n)) - cut
return optimize.brentq(g, 1e-12, p_hat), optimize.brentq(g, p_hat, 1 - 1e-12)
for k, n in [(7, 10), (70, 100)]:
p_hat = k / n
se = np.sqrt(p_hat * (1 - p_hat) / n)
lo, hi = lr_interval(k, n)
print(f"{k:2d} of {n:3d}: likelihood-ratio 95% interval {lo:.3f} to {hi:.3f};"
f" Wald interval {p_hat - 1.96 * se:.3f} to {p_hat + 1.96 * se:.3f}")Two features of maximum likelihood make it the default estimator, under regularity conditions (a smooth log-likelihood, a parameter inside its range rather than on a boundary, a number of parameters that does not grow with ).
Theorem— Large-sample behaviour of the MLE
As , is consistent (it converges to the true ) and asymptotically normal: approximately , so , with the inverse information matrix in several dimensions. Its asymptotic variance attains the Cramér–Rao lower bound, the smallest possible for unbiased estimators under the same conditions. And the MLE is invariant: the MLE of is .
For the seeds, , so , the familiar formula behind the Wald interval . The program also computes a likelihood-ratio interval: all values of whose log-likelihood is within of the maximum (half of 3.84, the 95% point of , for reasons the next section explains). With 7 of 10 seeds the Wald interval, 0.416 to 0.984, is symmetric and reaches almost to 1; the likelihood-ratio interval, 0.393 to 0.915, follows the asymmetric shape of the likelihood. With 70 of 100, the two nearly agree (0.606–0.784 and 0.610–0.790), because the log-likelihood has become nearly quadratic, which is exactly what asymptotic normality means.
For the normal linear model, maximising the likelihood over means minimising : least squares is maximum likelihood when errors are normal, and the logistic and Poisson fits of the previous lesson were maximum likelihood with other distributions. The deviance used there is times a log-likelihood ratio.
Maximum likelihood by numerical optimisation
Most likelihoods have no closed-form maximum, and we maximise them numerically. A classic case comes from wind energy. Engineers assessing a site in, say, the Atacama or on the coast of Tamil Nadu summarise a year of hourly wind speeds by a Weibull distribution, with density for : a scale near the typical speed and a shape describing how variable the wind is. The fitted distribution feeds the estimate of energy output, which depends heavily on the frequency of strong winds.
The program fits a Weibull distribution to 300 synthetic hourly speeds by minimising the negative log-likelihood with SciPy's BFGS optimiser. Two practical habits are visible. The parameters are optimised on the log scale, so the optimiser cannot wander into negative shapes or scales. And the answer is checked against an independent implementation, scipy.stats.weibull_min.fit. Standard errors come from the observed information, the matrix of second derivatives of the negative log-likelihood at the optimum, computed by finite differences. Finally, the program tests two simpler models nested inside the Weibull: the exponential distribution () and the Rayleigh distribution (), which wind engineers often assume when data are scarce.
import numpy as np
from scipy import stats, optimize
# 300 synthetic hourly mean wind speeds (m/s) at a candidate wind-farm site.
rng = np.random.default_rng(2024)
v = stats.weibull_min.rvs(2.1, scale=7.5, size=300, random_state=rng)
def nll(theta):
"""Negative log-likelihood of the Weibull(shape k, scale c), with theta = (log k, log c) so both stay positive."""
k, c = np.exp(theta)
return -np.sum(np.log(k / c) + (k - 1) * np.log(v / c) - (v / c) ** k)
res = optimize.minimize(nll, x0=[0.0, np.log(v.mean())], method="BFGS")
k_hat, c_hat = np.exp(res.x)
print(f"our MLE: shape k = {k_hat:.3f}, scale c = {c_hat:.3f} m/s (log-likelihood {-res.fun:.3f})")
k_sp, _, c_sp = stats.weibull_min.fit(v, floc=0)
print(f"SciPy fit: shape k = {k_sp:.3f}, scale c = {c_sp:.3f} m/s")
# Standard errors from the observed information: the Hessian of nll in (k, c), by central differences.
def nll_kc(p):
return nll(np.log(p))
p0, h = np.array([k_hat, c_hat]), 1e-4
H = np.zeros((2, 2))
for i in range(2):
for j in range(2):
ei, ej = np.eye(2)[i] * h, np.eye(2)[j] * h
H[i, j] = (nll_kc(p0 + ei + ej) - nll_kc(p0 + ei - ej) - nll_kc(p0 - ei + ej) + nll_kc(p0 - ei - ej)) / (4 * h * h)
se = np.sqrt(np.diag(np.linalg.inv(H)))
print(f"standard errors: k {se[0]:.3f}, c {se[1]:.3f}; 95% CI for k: {k_hat - 1.96 * se[0]:.2f} to {k_hat + 1.96 * se[0]:.2f}")
# Likelihood-ratio tests of two simpler models nested in the Weibull
ll_full = -res.fun
for name, k0 in [("exponential (k = 1)", 1.0), ("Rayleigh (k = 2)", 2.0)]:
fixed = optimize.minimize_scalar(lambda logc: nll([np.log(k0), logc]), bounds=(-5, 5), method="bounded")
lr = 2 * (ll_full + fixed.fun)
print(f"{name:>20}: LR = {lr:7.2f} on 1 df, p = {stats.chi2.sf(lr, 1):.2g}")
print(f"fitted P(v > 3 m/s, a typical cut-in speed) = {np.exp(-(3 / c_hat) ** k_hat):.3f}; observed {np.mean(v > 3):.3f}")Our optimiser and SciPy agree: and m/s, with standard errors 0.093 and 0.211. The exponential model is rejected overwhelmingly (likelihood ratio 188.10), while the Rayleigh model fits about as well as the full Weibull (): with these data, the simpler model is good enough. The fitted distribution says the wind exceeds 3 m/s, a typical turbine cut-in speed, 85.0% of the time, against 86.0% in the data.
Numerical maximum likelihood has failure modes worth knowing: an optimiser may stop at a local maximum (try several starting points), the likelihood may be flat in some direction because two parameters trade off against each other (the model is barely identifiable, and standard errors explode), or the maximum may sit on the boundary of the parameter space. The theory of optimisation in Lumenary Math explains when such methods converge.
Likelihood-ratio tests and Wilks' theorem
Comparing a model with a simpler one nested inside it (obtained by fixing some parameters) is the job of the likelihood-ratio test. The bigger model always has the higher maximised likelihood; the question is whether the gain is more than chance would produce.
Theorem— Wilks' theorem
Suppose model , with free parameters, is nested in model , with , and is true. Under regularity conditions, as ,
in distribution.
The idea of the proof is a Taylor expansion: near its maximum the log-likelihood is approximately quadratic, the MLE approximately normal, and twice the drop in a quadratic from its maximum, along directions, is a sum of squared standard normal variables. Samuel Wilks proved it in 1938. The theorem also justifies the likelihood-ratio interval above (the values of that such a test would not reject), and the deviance tests of the previous lesson. Is the approximation good at realistic sample sizes? The program simulates 1000 samples of 50 wind speeds from a Rayleigh distribution, so that the simpler model is true, and compares the likelihood-ratio statistic with .
import numpy as np
from scipy import stats, optimize
# Wilks' theorem by simulation: when the simpler model is true, is 2 * (gain in log-likelihood) ~ chi-squared(1)?
rng = np.random.default_rng(1938)
def weibull_loglik(k, c, v):
return np.sum(np.log(k / c) + (k - 1) * np.log(v / c) - (v / c) ** k)
def lr_statistic(v):
full = optimize.minimize(lambda t: -weibull_loglik(*np.exp(t), v), x0=[np.log(2.0), np.log(v.mean())], method="BFGS")
c_rayleigh = np.sqrt(np.mean(v ** 2)) # the MLE of the scale when the shape is fixed at 2
return 2 * (-full.fun - weibull_loglik(2.0, c_rayleigh, v))
lr = np.array([lr_statistic(stats.weibull_min.rvs(2.0, scale=7.5, size=50, random_state=rng)) for _ in range(1000)])
print(f"{'quantile':>8} {'simulated':>10} {'chi-squared(1)':>15}")
for q in [0.5, 0.9, 0.95, 0.99]:
print(f"{q:8.2f} {np.quantile(lr, q):10.2f} {stats.chi2.ppf(q, 1):15.2f}")
print(f"share of LR > 3.84 (a 5% test): {np.mean(lr > stats.chi2.ppf(0.95, 1)):.1%}")AIC and BIC
Likelihood-ratio tests compare two nested models. Often we have many candidate models, not all nested, and the goal is not to test a hypothesis but to choose a model that will serve well. The maximised log-likelihood cannot choose on its own, because it always favours the biggest model: each extra parameter can only raise it.
Hirotugu Akaike, at the Institute of Statistical Mathematics in Tokyo, asked in the early 1970s how well a fitted model would predict new data from the same process, measured by the expected log-likelihood of new data, which is related to the Kullback–Leibler divergence from the truth (see information theory). The maximised log-likelihood is an optimistic estimate of that, because the parameters were tuned to the data in hand, and he showed that the optimism is approximately , the number of estimated parameters. Correcting for it, and multiplying by by convention, gives
The Bayesian information criterion, proposed by Gideon Schwarz in 1978, comes from a different question: which model has the highest posterior probability, with equal prior probabilities on the models? For large , approximates the log of the marginal likelihood of the data under the model (Bayesian inference explains marginal likelihoods). Smaller is better for both, and only differences matter: a difference in AIC of about 2 or less is weak evidence, while 10 or more is strong.
The two criteria want different things. BIC's penalty, per parameter, exceeds AIC's 2 once , so BIC chooses simpler models. If one candidate is the true model, BIC finds it with probability tending to 1 (it is consistent), while AIC keeps a positive chance of choosing a model that is too big. But the true model is rarely among the candidates. When the goal is prediction and reality is more complex than any candidate, AIC's willingness to add parameters as data accumulate is an advantage, and it is asymptotically efficient for prediction. Neither is "right"; they answer different questions. For small samples, the corrected is preferable.
Both criteria compare models of the same data: the same observations, the same response on the same scale. Comparing the AIC of a model for rent with that of a model for log rent is meaningless without a correction for the transformation.
Cross-validation and overfitting
The most direct way to estimate how a model will do on new data is to hold some data back. In -fold cross-validation the data are split at random into parts (folds); each part in turn is left out, the model is fitted to the other , and its prediction error is measured on the left-out part. The average over folds estimates the out-of-sample error. Leave-one-out cross-validation (LOO-CV) is the case . For linear least squares it needs no refitting at all: the residual that observation would have if it were left out is , with the leverage from the hat matrix, so
Cross-validation assumes almost nothing about the model (it works for neural networks and random forests as well as for regressions), and Mervyn Stone showed in 1977 that LOO-CV and AIC are asymptotically equivalent for comparing models. The price is computation, and some care, discussed below, about what counts as "new" data.
Here is overfitting in action. Twenty noisy points come from a smooth curve, with noise standard deviation 0.3, and polynomials of degree 0 to 10 are fitted to them. Because the data are synthetic, we can also measure each fitted curve's true error on 5000 fresh points.
import numpy as np
# 20 noisy observations of a smooth curve; fit polynomials of degree 0 to 10.
rng = np.random.default_rng(1974)
f = lambda x: np.sin(3 * x)
n, sigma = 20, 0.3
x = rng.uniform(-1, 1, n)
y = f(x) + rng.normal(0, sigma, n)
x_new = rng.uniform(-1, 1, 5000) # fresh data from the same process, for the "truth"
y_new = f(x_new) + rng.normal(0, sigma, 5000)
folds = rng.permutation(n) % 5 # 5-fold cross-validation, same folds for every degree
rows = []
for d in range(11):
X = np.vander(x, d + 1)
b = np.linalg.lstsq(X, y, rcond=None)[0]
sse = np.sum((y - X @ b) ** 2)
k = d + 2 # d + 1 coefficients plus the error variance
loglik = -n / 2 * (np.log(2 * np.pi * sse / n) + 1) # maximised Gaussian log-likelihood
h = np.diag(X @ np.linalg.pinv(X)) # leverages, for the leave-one-out shortcut
loo = np.mean(((y - X @ b) / (1 - h)) ** 2)
cv5 = 0.0
for fold in range(5):
tr, te = folds != fold, folds == fold
bf = np.linalg.lstsq(X[tr], y[tr], rcond=None)[0]
cv5 += np.sum((y[te] - X[te] @ bf) ** 2) / n
test = np.mean((y_new - np.vander(x_new, d + 1) @ b) ** 2)
rows.append((d, sse / n, test, -2 * loglik + 2 * k, -2 * loglik + k * np.log(n), loo, cv5))
print(f"{'degree':>6} {'train MSE':>9} {'test MSE':>9} {'AIC':>6} {'BIC':>6} {'LOO-CV':>8} {'5-fold':>8}")
for r in rows:
print(f"{r[0]:6d} {r[1]:9.4f} {r[2]:9.4f} {r[3]:6.1f} {r[4]:6.1f} {r[5]:8.4g} {r[6]:8.4g}")
R = np.array(rows)
for j, name in [(1, "training error"), (2, "test error"), (3, "AIC"), (4, "BIC"), (5, "LOO-CV"), (6, "5-fold CV")]:
print(f"{name:>14} picks degree {int(R[np.argmin(R[:, j]), 0])}")The training error falls with every added degree, from 0.557 to 0.056: by that measure the degree-10 polynomial is the best model, and it is the worst. Its error on new data is 0.950, nine times the best achievable here. The true error is smallest at degree 3 (0.102, close to the noise floor of 0.09), and AIC, BIC, leave-one-out and 5-fold cross-validation all choose degree 3, without ever seeing the 5000 new points. Notice also how the cross-validation errors explode for high degrees (to 347.5 for 5-fold at degree 10): leaving out a point near the edge of the data forces a high-degree polynomial to extrapolate, and it swings wildly. That is overfitting made visible.
Two rules keep cross-validation honest. Everything that learns from the data must happen inside the folds: choosing variables, scaling, imputing missing values or tuning settings on the whole dataset, then cross-validating, leaks information from the held-out folds and makes the estimate optimistic. And the folds must respect the structure of the data: with repeated measurements of the same patients, leave out whole patients; with time series, never train on the future to predict the past. If a model's settings are tuned by cross-validation, an honest estimate of its performance needs a second, outer layer (nested cross-validation) or a test set locked away until the end.
Parsimony and its limits
Every method in this lesson rewards parsimony: a model should be no more complicated than the data can support. This is a statistical version of Occam's razor, and it has a precise justification, the bias–variance trade-off. A simple model may be biased, missing real structure; a complex one has low bias but high variance, because its many parameters are each estimated from limited information and chase the noise. The best predictions balance the two, and how much complexity the data can support grows with the amount of data.
George Box put the attitude in a famous phrase in 1979: all models are wrong, but some are useful. Model selection does not find the true model; it finds a useful one for a stated purpose. Several limits follow, and they are where good statisticians earn their keep.
- Selection invalidates naive inference. If you try many models and report the standard errors and -values of the one you chose, those numbers are too optimistic, because they ignore the selection. This is one of Andrew Gelman and Eric Loken's "forking paths": even without deliberate fishing, analysis choices that depend on the data make -values unreliable. The remedies are to pre-register the analysis, to split the data (select on one part, infer on the other), to average over models, or to use methods of post-selection inference designed for the purpose. See multiple testing and reproducibility.
- The best predictive model is not the best explanatory one. A model chosen by AIC or cross-validation to predict well may drop a confounder that matters for causal interpretation, or keep a proxy that predicts without explaining. Selection criteria answer "what predicts?", not "what causes?" (causal inference).
- Sometimes there is no single best model. When several models are within a couple of AIC units of each other, choosing one and discarding the rest understates uncertainty. Model averaging, weighting the predictions of several models, often predicts better and is more honest.
- Shrinkage is often better than selection. Rather than including or excluding each parameter, regularisation (ridge regression, the lasso) keeps many parameters but shrinks them towards zero, choosing the amount of shrinkage by cross-validation; this is the main tool when there are many predictors, in high-dimensional statistics.
- Modern machine learning stretches the classical picture. Deep neural networks often have far more parameters than data points, fit the training data exactly, and still predict well, a phenomenon studied under names such as "benign overfitting" and "double descent". The bias–variance trade-off is not repealed, but parameter count turns out to be a poor measure of complexity for such models. Statistical learning theory takes this up.
The professional view
In R, logLik(), AIC() and BIC() work for almost every fitted model, anova(fit0, fit1, test = "LRT") performs likelihood-ratio tests between nested GLMs, and optim() maximises custom likelihoods. In Python, statsmodels results carry .llf, .aic and .bic, SciPy's optimize.minimize handles custom likelihoods, and scikit-learn provides cross-validation splitters, including grouped and time-series splits, and tools for nested cross-validation. In Bayesian work with Stan and similar tools, approximate leave-one-out cross-validation by Pareto-smoothed importance sampling, implemented in the loo package, has largely replaced older information criteria; that is the subject of the Markov chain Monte Carlo lesson.
In applied work, the most important decisions come before any criterion is computed: which candidate models are scientifically plausible, what the model is for (prediction, explanation, a decision), and how its performance will be judged on data it has never seen. In regulated fields such as medical devices, credit scoring and official statistics, the validation plan is usually written down before the model is built, and the final test data are kept away from the modellers.
Common misconceptions
Misconception
"The likelihood is the probability that the parameter is correct." is the probability of the data under , read as a function of . It does not integrate to 1 over , and turning it into probabilities about requires a prior distribution, which is Bayesian inference.
Misconception
"The model with the lowest AIC is the true model." AIC estimates relative predictive accuracy among the candidates you supplied. All of them may be wrong, and models within about 2 units are practically indistinguishable.
Misconception
"Cross-validation error measures how the final model will do in deployment." Only if the folds mimic deployment. Random folds on time-ordered data, or on repeated measurements of the same people, or after selecting features on all the data, give optimistic estimates. And a model chosen as the best of many by cross-validation is itself optimistically evaluated by that same cross-validation.
Misconception
"Overfitting means the model is too big." Overfitting means fitting noise, and it depends on the model's flexibility relative to the information in the data. Twenty points cannot support a degree-10 polynomial; twenty thousand could. And a large model with suitable regularisation can generalise well.
Exercises
1. Likelihood and model selection
Warm-upHumanIn the lesson, 7 of 10 seeds germinated, and . Which statement is correct?
2. Likelihood and model selection
Warm-upHumanA clinic in Kampala records the number of patients arriving with malaria on five Mondays: 3, 5, 4, 0, 8. Modelling the counts as independent Poisson(), what is the maximum-likelihood estimate of ?
3. Likelihood and model selection
CoreHumanTwo models are fitted to the same 100 observations. Model A has 3 parameters and maximised log-likelihood ; model B has 5 parameters and maximised log-likelihood . What is the AIC of model B? (Then compare it with A's.)
4. Likelihood and model selection
CoreHumanA Poisson regression of hospital admissions has 4 coefficients; adding 3 air-pollution measures gives a model with 7. The maximised log-likelihoods are and . What does a likelihood-ratio test at the 5% level conclude? (The 95% point of is 7.81.)
5. Likelihood and model selection
CoreHumanAn analyst has 80 patients and 5000 gene-expression measurements. She first picks the 20 genes most correlated with the outcome using all 80 patients, then fits a model on those 20 genes and estimates its accuracy by 10-fold cross-validation, getting 90%. What is the main problem?
6. Likelihood and model selection
StretchHumanWhich statements about AIC and BIC are correct? Select all that apply.
7. Likelihood and model selection
CoreComputationalRainfall amounts on wet days are often modelled by a gamma distribution with shape and scale , density for . Write fit_gamma(x) that returns the maximum-likelihood estimates (a, s) by minimising the negative log-likelihood with scipy.optimize.minimize. Optimise over so that both stay positive, and use scipy.special.gammaln for . The tests compare your answer with scipy.stats.gamma.fit(x, floc=0).
8. Likelihood and model selection
StretchComputationalWrite loocv_mse(X, y) that returns the leave-one-out cross-validation mean squared error of a least-squares fit without refitting times, using the shortcut from the lesson: , where are the ordinary residuals and the leverages (the diagonal of ). The tests compare your answer with brute-force refits.
Further reading and resources
- SciPy: optimisation (scipy.optimize)The minimisers and root finders used in this lesson.
- SciPy: statistical functions (scipy.stats)Every distribution has a fit method for maximum likelihood.
- scikit-learn: cross-validationk-fold, grouped and time-series cross-validation, and nested cross-validation for tuned models.
- A new look at the statistical model identification · Hirotugu Akaike · 1974IEEE Transactions on Automatic Control, volume 19. The paper that introduced AIC to a wide audience.
- Estimating the dimension of a model · Gideon Schwarz · 1978Annals of Statistics, volume 6. The Bayesian information criterion.
- The Elements of Statistical Learning · Trevor Hastie, Robert Tibshirani and Jerome FriedmanFree online. Its chapter on model assessment and selection covers AIC, BIC, cross-validation and the bias–variance trade-off in depth.
- loo: efficient leave-one-out cross-validation for Bayesian modelsThe Stan project's package for approximate leave-one-out cross-validation, with its documentation and papers.
- The garden of forking paths · Andrew Gelman and Eric Loken · 2013Why data-dependent choices of analysis invalidate p-values, even without any deliberate fishing.