Which schools are best?
A regional education office publishes a league table of its 40 primary schools, ranked by the average score of their pupils on a mathematics test. The top of the table is crowded with very small schools. So is the bottom. Are small schools somehow both better and worse than large ones?
No. A school's average is an estimate, and its precision depends on how many pupils it rests on. With pupils' scores varying by about 60 points around their school's mean, an average of 5 pupils has a standard error of points; an average of 180 pupils, about 4.5. Small schools land at the extremes of the table because their averages are noisy, not because they are extreme. Howard Wainer has called the formula behind this, , "the most dangerous equation", and argued that ignoring it helped motivate large investments in small schools in the United States around 2000, on the strength of their presence among top performers. The same trap catches rankings of hospitals by surgical mortality, of districts by disease rates, and of sales staff by monthly results.
This lesson is about data with groups: pupils within schools, patients within hospitals, repeated measurements within people, survey respondents within regions, trials within a meta-analysis. Grouped data raise two problems. Estimating each group's mean separately is noisy for small groups, and treating all observations as independent overstates how much we know. Multilevel models (also called hierarchical models, mixed models or random-effects models) solve both by modelling the groups themselves as draws from a population.
Three ways to estimate a school's mean
Suppose we want the best estimate of each school's true average score. There are three natural choices.
- Complete pooling ignores the schools: every school is estimated by the overall mean. Stable, but it pretends schools do not differ.
- No pooling uses each school's own mean. Unbiased for each school, but very noisy for small ones.
- Partial pooling uses a weighted compromise: a school's own mean, pulled towards the overall mean by an amount that depends on how much its own data can be trusted. Large schools are barely moved; small ones are moved a lot.
Analogy, not a formal construction
Two cafés on a street in Lagos have online ratings. One has 4.9 stars from 3 reviews; the other has 4.6 stars from 600. Most people would guess that the second is the better café, because 3 glowing reviews could easily come from the owner's friends, and a typical café on that street rates about 4.2. Without writing any formula, they have shrunk the small café's rating towards the street's average, and left the large café's almost alone.
Partial pooling is the statistical version of that judgement, with the amount of shrinkage learned from the data: from how much schools really differ, compared with how much noise sits in each school's average.
The random-intercept model
Definition— random-intercept model
Let be the score of pupil in school , for schools with pupils each. The random-intercept model is with all terms independent. Here is the variance within schools, the variance of true school means between schools, and the intraclass correlation is the share of total variance that lies between schools, which is also the correlation between two pupils of the same school.
Predictors enter as in any regression: adds a pupil-level predictor such as prior attainment, and a school-level predictor such as funding enters the model for . Letting the slope vary too, with drawn from a bivariate normal distribution, gives a random-slopes model, in which the effect of prior attainment can differ between schools. The second level is itself a model, so it can have its own predictors, levels can be stacked (pupils in classes in schools in districts), and the outcome can be binary or a count, giving generalised linear mixed models.
Theorem— Shrinkage of group means
In the random-intercept model with , and known, the best estimate of given the data (its posterior mean, and the best linear unbiased predictor) is where is school 's own mean.
Proof
Given , the school mean is , and the prior is . This is the normal–normal model of Bayesian inference: the posterior of is normal with precision and a mean that is the precision-weighted average of and . The weight on is , which simplifies to .
∎
The weight compares two variances: the real spread of schools () and the noise in this school's average (). When is large the noise is small, , and the school keeps its own mean. When is small, or when schools barely differ, and the estimate moves to . In practice , and are unknown and estimated from all schools together, by maximum likelihood, by restricted maximum likelihood (REML, which corrects the downward bias of the variance estimates), or by placing priors on them and computing the full posterior. Plugging estimates into the formula is called empirical Bayes.
Partial pooling in action
The simulation creates a region of 40 schools with between 5 and 200 pupils each, true school means drawn from and pupils scattered around their school's mean with standard deviation 60. It estimates by maximising the marginal likelihood of the school means, each of which is , and then compares the three kinds of pooling against the true means, which a simulation, unlike reality, lets us see.
import numpy as np
from scipy.optimize import minimize_scalar
rng = np.random.default_rng(2024)
J = 40
size = np.round(np.exp(rng.uniform(np.log(5), np.log(200), J))).astype(int) # 5 to 200 pupils per school
true_mean = rng.normal(500, 15, J) # each school's true average score
school = np.repeat(np.arange(J), size)
score = rng.normal(true_mean[school], 60) # pupils vary much more than schools do
ybar = np.bincount(school, score) / size # no pooling: each school's own mean
grand = score.mean() # complete pooling: one mean for all
resid = score - ybar[school]
sigma2 = resid @ resid / (len(score) - J) # pooled within-school variance
def neg_marginal_loglik(log_tau):
"""Each school mean is N(mu, tau^2 + sigma^2/n_j); mu is profiled out by weighted averaging."""
v = np.exp(2 * log_tau) + sigma2 / size
mu = np.sum(ybar / v) / np.sum(1 / v)
return 0.5 * np.sum(np.log(v) + (ybar - mu) ** 2 / v)
tau2 = np.exp(2 * minimize_scalar(neg_marginal_loglik, bounds=(-5, 6), method="bounded").x)
v = tau2 + sigma2 / size
mu = np.sum(ybar / v) / np.sum(1 / v)
weight = tau2 / (tau2 + sigma2 / size) # how much each school's own mean is trusted
partial = mu + weight * (ybar - mu) # partial pooling (empirical Bayes)
print(f"estimated within-school sd {np.sqrt(sigma2):.1f} (true 60), between-school sd {np.sqrt(tau2):.1f} (true 15)")
print(f"intraclass correlation {tau2 / (tau2 + sigma2):.3f}")
print("school pupils own mean weight partial pooling true mean")
for j in np.argsort(size)[[0, 1, 2, -3, -2, -1]]:
print(f"{j:5d} {size[j]:6d} {ybar[j]:8.1f} {weight[j]:6.2f} {partial[j]:13.1f} {true_mean[j]:9.1f}")
for name, est in [("no pooling", ybar), ("complete pooling", np.full(J, grand)), ("partial pooling", partial)]:
print(f"{name:17} root mean squared error against the true means: {np.sqrt(np.mean((est - true_mean) ** 2)):5.2f}")
top = np.argsort(-ybar)[:5]
print("top five schools by raw mean have", size[top], "pupils; true top five have", size[np.argsort(-true_mean)[:5]])The model recovers the two variances well: 61.0 within schools and 14.6 between them, an intraclass correlation of only 0.054. The table shows shrinkage at work. School 29, with 6 pupils, has its own mean of 521.6 but a weight of only 0.25, so its estimate is pulled to 506.2 (its true mean is 513.3). School 4, with 197 pupils, keeps 92% of its distance from the mean: 468.0 becomes 470.7, close to the true 468.5. Overall, partial pooling cuts the error from 12.07 points (no pooling) to 7.54, and beats complete pooling (13.74) too. Four of the five schools at the top of the raw league table have fewer than 20 pupils.
One region could be lucky. The next block repeats the whole experiment for 1,000 regions.
import numpy as np
from scipy.optimize import minimize_scalar
rng = np.random.default_rng(1961)
J, reps = 40, 1000
rmse = {"no pooling": [], "partial pooling": []}
top_small_raw = top_small_true = top_small_partial = 0
for r in range(reps):
size = np.round(np.exp(rng.uniform(np.log(5), np.log(200), J))).astype(int)
true_mean = rng.normal(500, 15, J)
school = np.repeat(np.arange(J), size)
score = rng.normal(true_mean[school], 60)
ybar = np.bincount(school, score) / size
resid = score - ybar[school]
sigma2 = resid @ resid / (len(score) - J)
def nll(log_tau):
v = np.exp(2 * log_tau) + sigma2 / size
mu = np.sum(ybar / v) / np.sum(1 / v)
return 0.5 * np.sum(np.log(v) + (ybar - mu) ** 2 / v)
tau2 = np.exp(2 * minimize_scalar(nll, bounds=(-5, 6), method="bounded").x)
v = tau2 + sigma2 / size
mu = np.sum(ybar / v) / np.sum(1 / v)
partial = mu + tau2 / v * (ybar - mu)
rmse["no pooling"].append(np.sqrt(np.mean((ybar - true_mean) ** 2)))
rmse["partial pooling"].append(np.sqrt(np.mean((partial - true_mean) ** 2)))
small = size < 20
top_small_raw += small[np.argmax(ybar)]
top_small_partial += small[np.argmax(partial)]
top_small_true += small[np.argmax(true_mean)]
print(f"{reps} simulated regions of {J} schools")
for k, v in rmse.items():
print(f" {k:15} average RMSE {np.mean(v):5.2f}")
print(f" partial pooling more accurate in {np.mean(np.array(rmse['partial pooling']) < np.array(rmse['no pooling'])):.0%} of regions")
print(f"share of regions whose top-ranked school has fewer than 20 pupils:")
print(f" by raw mean {top_small_raw / reps:.0%} by partial pooling {top_small_partial / reps:.0%} truly best {top_small_true / reps:.0%}")Across 1,000 regions, partial pooling reduces the average error from 13.60 to 9.23 points and is more accurate in 99% of them. The ranking lines tell a subtler story. About 35% of the time the truly best school is a small one (roughly the share of small schools), but the raw league table puts a small school at the top 66% of the time. Partial pooling over-corrects: only 9%. A small school must look spectacular before its estimate escapes the pull of the mean.
Theorem— Stein's paradox (James and Stein, 1961)
If are independent with , known and , then the estimator that shrinks each towards zero by the factor has smaller expected total squared error than itself, whatever the true values .
This result shocked statisticians in 1961: when estimating three or more unrelated quantities at once, the obvious estimate is beaten by one that borrows strength across them. Bradley Efron and Carl Morris made it famous by predicting baseball batting averages for the rest of a season from the first 45 at-bats; the shrunken estimates were far closer to the final averages. Multilevel models are the practical form of the paradox: the shrinkage target is the group mean rather than zero, and the amount comes from the estimated .
The price of ignoring the groups
Shrinkage is about estimating groups. The second reason for multilevel models matters even when nobody cares about individual groups: observations in the same group are correlated, so they carry less information than the same number of independent observations. Ignoring that makes standard errors too small and -values too optimistic.
Consider a cluster-randomised trial: 20 schools, 10 of them chosen at random to try a new curriculum, 30 pupils tested in each. The unit that was randomised is the school. The simulation below gives the curriculum no effect at all, sets the intraclass correlation to 0.15 (a typical value for test scores), and runs 4,000 such trials.
import numpy as np
from scipy import stats
rng = np.random.default_rng(1986)
schools, pupils, reps = 20, 30, 4000 # 10 schools try a new curriculum, 10 do not
icc = 0.15 # share of score variance that lies between schools
tau, sigma = np.sqrt(icc), np.sqrt(1 - icc)
# reps simulated trials at once. The curriculum does NOTHING: only school effects and pupil noise.
y = rng.normal(0, tau, (reps, schools, 1)) + rng.normal(0, sigma, (reps, schools, pupils))
a, b = y[:, :10], y[:, 10:] # treated and control schools
def two_sample_p(x, z):
"""Pooled-variance two-sample t-test along the last axis, for many data sets at once."""
nx, nz = x.shape[-1], z.shape[-1]
sp2 = ((nx - 1) * x.var(axis=-1, ddof=1) + (nz - 1) * z.var(axis=-1, ddof=1)) / (nx + nz - 2)
t = (x.mean(axis=-1) - z.mean(axis=-1)) / np.sqrt(sp2 * (1 / nx + 1 / nz))
return 2 * stats.t.sf(np.abs(t), df=nx + nz - 2)
# Wrong: treat all 600 pupils as independent observations.
p_pupil = two_sample_p(a.reshape(reps, -1), b.reshape(reps, -1))
# Right (simplest version): analyse one mean per school, the unit that was randomised.
p_school = two_sample_p(a.mean(axis=2), b.mean(axis=2))
print(f"false-positive rate, pupils as independent: {np.mean(p_pupil < 0.05):.1%}")
print(f"false-positive rate, school means: {np.mean(p_school < 0.05):.1%}")
deff = 1 + (pupils - 1) * icc
print(f"design effect 1 + (m - 1) * ICC = {deff:.2f}: "
f"{schools * pupils} clustered pupils carry the information of about {schools * pupils / deff:.0f} independent ones")Analysing the 600 pupils as if they were independent declares a "significant" effect of a useless curriculum in 39.8% of trials, eight times the advertised 5%. Analysing one mean per school, the unit that was randomised, gives 5.4%, consistent with 5% given the simulation's own noise. The design effect, for clusters of size , explains the gap: with and it is 5.35, so the 600 pupils are worth about 112 independent ones. A mixed model with a random intercept for school reaches the same conclusion as the school-means analysis here, and also handles unequal cluster sizes and pupil-level covariates.
Fixed effects, random effects, and when to use mixed models
The words "fixed" and "random" cause more confusion than anything else in this subject, because different fields use them differently; Andrew Gelman counted five incompatible definitions in the literature. Two matter most.
- In statistics, a random effect is a set of group-level parameters given a distribution (), so that they are partially pooled. A fixed effect is a parameter estimated without such a distribution.
- In econometrics, a "fixed-effects model" includes a separate intercept for every group (no pooling), and the point is usually not to estimate those intercepts but to remove every stable difference between groups from the estimate of a slope. A "random-effects model" pools the intercepts, and is valid for the slope only if the group effects are uncorrelated with the predictors.
That last condition is the real substance behind the jargon. Suppose schools with richer families also have better teachers, and we model pupils' scores on family income with random school intercepts. The school effects are then correlated with income, and the random-effects estimate of the income slope mixes the within-school effect of income with the between-school difference in teachers. Econometricians use the Hausman test to compare the two estimates; a simpler and widely recommended remedy, due to Yair Mundlak, is to add the school mean of the predictor as a school-level predictor, which separates the within-school and between-school slopes inside one multilevel model.
Mixed models are the natural choice when:
- observations come in groups, and you want either estimates for the groups (schools, hospitals, regions) or valid inference about effects at the population level;
- there are enough groups to estimate the between-group variance: with fewer than about five to ten groups, is poorly determined, and either fixed group intercepts or a Bayesian analysis with an informative prior on is safer;
- group sizes are unequal, so that some groups need more pooling than others;
- there are repeated measurements on the same people or units over time (growth curves, longitudinal studies), which are groups too;
- results from several studies are combined: a random-effects meta-analysis treats each study's true effect as a draw from a distribution of effects, and is the standard in medical evidence synthesis;
- estimates are needed for many small areas: national statistical offices use multilevel models for small-area estimation, borrowing strength across districts to estimate poverty or unemployment where surveys reach few households (see Privacy and official statistics).
Mixed models in software
Fitting mixed models is routine in every statistical environment. In R the lme4 package's formula notation has become a lingua franca: (1 | school) means "a random intercept for each school", and (1 + prior | school) adds a random slope. The code below is not run here.
# R, not run here: random intercepts, then random slopes for prior attainment
library(lme4)
m1 <- lmer(score ~ prior + (1 | school), data = pupils, REML = TRUE)
m2 <- lmer(score ~ prior + (1 + prior | school), data = pupils)
summary(m1) # fixed effects, and the school and residual standard deviations
ranef(m1)$school # the partially pooled school effectsPython's statsmodels offers the same models (statsmodels is not loaded in the browser, so this is not run here):
# Python with statsmodels, not run here
import statsmodels.formula.api as smf
m1 = smf.mixedlm("score ~ prior", data=pupils, groups=pupils["school"]).fit(reml=True)
print(m1.summary())Bayesian fits put priors on , and and sample the posterior with Markov chain Monte Carlo (Markov chain Monte Carlo), which propagates the uncertainty in that empirical Bayes ignores. In Stan, the random-intercept model is a few lines (not run here):
// Stan, not run here
data { int<lower=1> N; int<lower=1> J; array[N] int<lower=1, upper=J> school; vector[N] y; }
parameters { real mu; real<lower=0> tau; real<lower=0> sigma; vector[J] z; }
transformed parameters { vector[J] alpha = mu + tau * z; } // non-centred form samples better
model {
mu ~ normal(500, 100); tau ~ normal(0, 50); sigma ~ normal(0, 100); z ~ std_normal();
y ~ normal(alpha[school], sigma);
}The R packages brms and rstanarm translate lme4-style formulas into Stan models, and Julia's MixedModels.jl is notably fast for large crossed designs. Whatever the software, read its warnings: a singular fit (an estimated variance of exactly zero, or a correlation of ±1) means the data cannot support the random-effects structure you asked for.
Where multilevel models came from
Mixed models grew up in several places at once. In the 1950s the American animal geneticist Charles Henderson developed the "mixed model equations" and best linear unbiased prediction to estimate the genetic merit of dairy bulls from their daughters' milk yields: a bull with three daughters is shrunk towards the population mean far more than one with three hundred, and breeding programmes worldwide still rest on the method. Charles Stein's 1956 discovery and the James–Stein estimator of 1961 gave shrinkage its theoretical foundation; Dennis Lindley and Adrian Smith's 1972 paper put hierarchical models into a Bayesian framework, and Nan Laird and James Ware's 1982 paper made random-effects models standard for longitudinal medical data. Education researchers, with Harvey Goldstein in London and Stephen Raudenbush and Anthony Bryk in the United States prominent among them, developed multilevel modelling for pupils in schools in the 1980s and wrote the software that spread it through the social sciences.
Common misconceptions
Misconception
"A random effect needs groups that were randomly sampled." Random effects are a modelling device for partial pooling, and they are routinely used for all 50 states of a country or all hospitals in a region. What they assume is exchangeability: before seeing the data, we have no reason to expect any particular group to differ from the others in a specific direction. If we do (urban versus rural schools), that information belongs in the model as a group-level predictor.
Misconception
"Shrinkage introduces bias, so it is cheating." Each shrunken estimate is biased towards the mean, but the reduction in variance more than pays for it: in the simulation, partial pooling was more accurate in 99% of regions. Unbiasedness for each group separately is a weaker goal than accuracy for the groups together.
Misconception
"With thousands of observations, clustering does not matter." Precision for group-level questions is limited by the number of groups. In the simulated cluster trial, 600 pupils carried the information of about 112, and ignoring the clustering produced a false positive in nearly 40% of trials.
Misconception
"The mixed model will sort out which effects should be random." The random-effects structure is a modelling choice driven by the design: which units were sampled or randomised, and which effects can plausibly vary between them. Data can show that a structure is too rich (a singular fit) but cannot choose the design for you.
Exercises
1. Multilevel models
Warm-upHumanIn a random-intercept model for hospital patient-satisfaction scores, the estimated overall mean is , the between-hospital standard deviation is , and the within-hospital standard deviation is . A hospital in Cebu has 16 survey responses with mean 530. What is its partially pooled (empirical Bayes) estimate?
2. Multilevel models
CoreHumanA survey in Peru interviews 21 households in each of 40 villages (840 households). For the outcome of interest, the intraclass correlation within villages is 0.1. What is the effective sample size, the number of independent households that would give the same precision for the overall mean?
3. Multilevel models
CoreHumanA trial in Bangladesh randomises 30 health clinics, half to a new hand-washing campaign, and records whether each of about 200 children per clinic has diarrhoea in the following month. Which analysis is appropriate?
4. Multilevel models
StretchHumanAn economist models workers' wages on years of training, with a random intercept for each firm. Firms that pay more for unobserved reasons (better management, say) also train their workers more. What problem does this cause, and what is a standard fix?
5. Multilevel models
CoreComputationalWrite partial_pool(ybar, n, mu, tau2, sigma2), which returns a NumPy array of partially pooled estimates with , for arrays (or lists) of group means ybar and group sizes n, and numbers mu, tau2 and sigma2.
6. Multilevel models
StretchComputationalFor balanced grouped data (a 2-D array y with one row per group and observations per row), the one-way ANOVA estimates are
- and ;
- and .
Write icc_anova(y) that returns the estimated intraclass correlation as a float.
Further reading and resources
- Data analysis using regression and multilevel/hierarchical models · Andrew Gelman and Jennifer HillThe book's web page, with data and code for its many multilevel examples.
- Centre for Multilevel Modelling, University of BristolFree online course materials on multilevel modelling, especially for education and social science.
- Data analysis using Stein's estimator and its generalizations · Bradley Efron and Carl Morris · 1975
- Stein's paradox in statistics · Bradley Efron and Carl Morris · 1977
- Bayes estimates for the linear model · Dennis V. Lindley and Adrian F. M. Smith · 1972
- Random-effects models for longitudinal data · Nan M. Laird and James H. Ware · 1982
- Analysis of variance — why it is more important than ever · Andrew Gelman · 2005Includes a discussion of the five incompatible definitions of "fixed" and "random" effects.
- The most dangerous equation · Howard Wainer · 2007An essay in American Scientist on how ignoring the standard error of small samples misleads policy, including the case of small schools.
- Funnel plots for comparing institutional performance · David J. Spiegelhalter · 2005
- On the pooling of time series and cross section data · Yair Mundlak · 1978
- Fitting linear mixed-effects models using lme4 · Douglas Bates, Martin Mächler, Ben Bolker and Steve Walker · 2015
- Prior distributions for variance parameters in hierarchical models · Andrew Gelman · 2006
- statsmodels: linear mixed effects models
- Stan documentation