Skip to content
LumenaryAcademy
Modelling 70 min

Linear regression

Fitting a straight line by least squares, reading what its slope and intercept say, measuring how sure we are, and the two classic traps, regression to the mean and extrapolation.

Code tested · 5AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

A line through a cloud of points

Chimwemwe is an agronomist working with smallholder maize farmers near Lilongwe, in Malawi. Fertiliser is expensive, and the question farmers put to her is practical: if I apply more nitrogen, how much more maize will I harvest? To answer it she runs a trial on one farm: 24 plots, eight rates of nitrogen from 0 to 140 kilograms per hectare, three plots at each rate, with the rates assigned to plots at random. At harvest she weighs the grain from each plot. (The numbers in this lesson are synthetic, generated by a program, but they behave like real trial data.)

Plotted, the 24 yields form a cloud that rises from left to right. No line passes through every point: plots that received the same nitrogen differ by half a tonne per hectare or more, because soil, water and pests vary from plot to plot. Statistics begins by accepting that variation. The question is not "which line passes through the points?" but "which line summarises how yield tends to change with nitrogen, how precisely do the data pin it down, and how far can we trust it?"

That is linear regression, the most widely used model in statistics. It is how researchers estimate how blood pressure changes with salt intake, how wages change with years of schooling, how a sensor's reading drifts with temperature. It is also the gateway to the rest of this level: multiple regression adds predictors, generalised linear models handle yes-or-no outcomes and counts, and analysis of variance turns out to be regression in disguise.

IntuitionH

What makes a line the best one?

A straight line is y^=b0+b1x\hat y = b_0 + b_1 x: an intercept b0b_0, the height of the line at x=0x = 0, and a slope b1b_1, the change in y^\hat y when xx increases by one unit. For any candidate line, each plot has a residual: its observed yield minus the yield the line predicts at its nitrogen rate, ei=yi−y^ie_i = y_i - \hat y_i. Points above the line have positive residuals, points below it negative ones.

A good line makes the residuals small overall, and the classic way to make "small overall" precise is to minimise the sum of squared residuals:

SSE(b0,b1)=∑i=1n(yi−b0−b1xi)2.\text{SSE}(b_0, b_1) = \sum_{i=1}^{n} (y_i - b_0 - b_1 x_i)^2.

Why vertical distances? Because the task is to predict yy from xx: the nitrogen rate was set by the experimenter and is known exactly, and the unpredictable part lives in the yield. Why squares? Squaring makes every miss count positively and punishes large misses more than small ones, and, decisively, it turns the minimisation into two linear equations, so the answer has a formula. Other choices exist (minimising absolute residuals gives a line that resists outliers, the start of quantile regression), but least squares is the default and the foundation of this level.

020406080100120140012345nitrogen applied, N (kg/ha)yield (t/ha)ŷ = 1.452 + 0.0202 N
Chimwemwe's synthetic trial: 24 plots, three at each nitrogen rate (shifted slightly sideways so they do not overlap). The blue line is the least-squares line; each orange segment is a residual, the vertical distance from a plot's yield to the line. Least squares chooses the line that makes the sum of the squared lengths of these segments as small as possible.
DefinitionsH

The model and the least-squares estimates

A line fitted to data is a description. To make inferences, about what would happen on other plots or in another season, we need a model: a statement about how the data were generated.

Definition— simple linear regression model

The responses Y1,…,YnY_1, \dots, Y_n, observed at known values x1,…,xnx_1, \dots, x_n of a predictor, satisfy

Yi=β0+β1xi+εi,Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i,

where the errors εi\varepsilon_i are independent, with mean 0 and the same variance σ2\sigma^2. For exact small-sample inference we add that they are normally distributed. The unknown parameters are β0\beta_0, β1\beta_1 and σ\sigma.

Greek letters stand for the truth we do not know; the Latin letters b0,b1b_0, b_1 (also written β^0,β^1\hat\beta_0, \hat\beta_1) are estimates computed from data. The assumptions are often remembered as LINE: a Linear mean, Independent errors, Normal errors, Equal variance. They are not equally important: a wrong shape for the mean, or dependent errors, can ruin the conclusions, while mild non-normality matters little once nn is moderate, thanks to the central limit theorem from sampling distributions.

Definition— fitted values and residuals

Given estimates b0,b1b_0, b_1, the fitted value of observation ii is y^i=b0+b1xi\hat y_i = b_0 + b_1 x_i and its residual is ei=yi−y^ie_i = y_i - \hat y_i. Residuals are the observable stand-ins for the unobservable errors εi\varepsilon_i.

Write xˉ,yˉ\bar x, \bar y for the means, and Sxx=∑(xi−xˉ)2S_{xx} = \sum (x_i - \bar x)^2, Sxy=∑(xi−xˉ)(yi−yˉ)S_{xy} = \sum (x_i - \bar x)(y_i - \bar y).

Theorem— Least-squares estimates

If the xix_i are not all equal, exactly one line minimises the sum of squared residuals. Its slope and intercept are

b1=SxySxx,b0=yˉ−b1xˉ.b_1 = \frac{S_{xy}}{S_{xx}}, \qquad b_0 = \bar y - b_1 \bar x.

Equivalently b1=r sy/sxb_1 = r\, s_y / s_x, where rr is the correlation and sx,sys_x, s_y the standard deviations. The line passes through the point of means (xˉ,yˉ)(\bar x, \bar y), and its residuals sum to zero and are uncorrelated with xx.

Proof

SSE is a sum of squares of linear functions of (b0,b1)(b_0, b_1), hence a smooth convex function, and its minimum is where both partial derivatives vanish:

∂ SSE∂b0=−2∑(yi−b0−b1xi)=0,∂ SSE∂b1=−2∑xi(yi−b0−b1xi)=0.\frac{\partial\,\text{SSE}}{\partial b_0} = -2\sum (y_i - b_0 - b_1 x_i) = 0, \qquad \frac{\partial\,\text{SSE}}{\partial b_1} = -2\sum x_i (y_i - b_0 - b_1 x_i) = 0.

The first equation says ∑ei=0\sum e_i = 0, which rearranges to b0=yˉ−b1xˉb_0 = \bar y - b_1 \bar x. The second says ∑xiei=0\sum x_i e_i = 0. Subtracting xˉ∑ei=0\bar x \sum e_i = 0 from it gives ∑(xi−xˉ)ei=0\sum (x_i - \bar x) e_i = 0, and substituting ei=(yi−yˉ)−b1(xi−xˉ)e_i = (y_i - \bar y) - b_1(x_i - \bar x) gives Sxy−b1Sxx=0S_{xy} - b_1 S_{xx} = 0. Since the xix_i are not all equal, Sxx>0S_{xx} > 0, so b1=Sxy/Sxxb_1 = S_{xy}/S_{xx}. The Hessian of SSE is 2(n∑xi∑xi∑xi2)2\begin{pmatrix} n & \sum x_i \\ \sum x_i & \sum x_i^2 \end{pmatrix}, with determinant 4nSxx>04 n S_{xx} > 0 and positive diagonal, so this stationary point is the unique minimum. Finally, r=Sxy/SxxSyyr = S_{xy}/\sqrt{S_{xx} S_{yy}} and sy/sx=Syy/Sxxs_y/s_x = \sqrt{S_{yy}/S_{xx}}, whose product is Sxy/SxxS_{xy}/S_{xx}; and ∑(xi−xˉ)ei=0\sum (x_i - \bar x) e_i = 0 is exactly zero covariance between xx and the residuals.

∎

The two conditions in the proof are the normal equations, and they have a geometric meaning worth carrying into the rest of the course. Stack the data as vectors in Rn\mathbb{R}^n: the responses y\mathbf y, and the columns of the design matrix X=[ 1    x ]X = [\,\mathbf 1 \;\; \mathbf x\,]. Every candidate line gives a vector of fitted values XbX\mathbf b, a point in the plane spanned by 1\mathbf 1 and x\mathbf x. Least squares picks the point of that plane closest to y\mathbf y, which is the orthogonal projection of y\mathbf y onto it: the residual vector e=y−Xb\mathbf e = \mathbf y - X\mathbf b is perpendicular to both columns, X⊤e=0X^\top \mathbf e = \mathbf 0. Solving gives

b=(X⊤X)−1X⊤y,\mathbf b = (X^\top X)^{-1} X^\top \mathbf y,

a formula that works unchanged when XX has more columns, as it will in the next lesson. Column spaces, projections and rank are covered in Lumenary Math's linear algebra.

ComputationC

Fitting the line in code

The program below generates the synthetic trial, fits the line from the formulas, and checks the answer against two library routines. It also prints the average residual at each nitrogen rate, which turns out to be the most interesting line of output.

python
import numpy as np
from scipy import stats

# Synthetic field trial: 24 maize plots, nitrogen fertiliser N (kg/ha), grain yield (t/ha).
rng = np.random.default_rng(2026)
N = np.repeat([0, 20, 40, 60, 80, 100, 120, 140], 3).astype(float)
true_curve = lambda n: 1.2 + 4.0 * (1 - np.exp(-n / 110))      # how this synthetic world works
yield_t = true_curve(N) + rng.normal(0, 0.35, N.size)

# Least squares from the formulas
xbar, ybar = N.mean(), yield_t.mean()
Sxx = np.sum((N - xbar) ** 2)
Sxy = np.sum((N - xbar) * (yield_t - ybar))
b1 = Sxy / Sxx
b0 = ybar - b1 * xbar
resid = yield_t - (b0 + b1 * N)
R2 = 1 - np.sum(resid ** 2) / np.sum((yield_t - ybar) ** 2)
print(f"slope     b1 = {b1:.4f} t/ha per kg/ha of N  ({1000 * b1:.1f} kg of grain per kg of N)")
print(f"intercept b0 = {b0:.3f} t/ha")
print(f"R^2 = {R2:.3f}   residuals sum to zero: {abs(resid.sum()) < 1e-9}")

# The same fit two other ways
X = np.column_stack([np.ones_like(N), N])
coef = np.linalg.lstsq(X, yield_t, rcond=None)[0]
fit = stats.linregress(N, yield_t)
print("np.linalg.lstsq:", np.round(coef, 4), "  scipy linregress:", round(fit.intercept, 4), round(fit.slope, 4))

# Mean residual at each dose: is a straight line the right shape?
print("N:            ", "  ".join(f"{n:5.0f}" for n in np.unique(N)))
print("mean residual:", "  ".join(f"{resid[N == n].mean():+5.2f}" for n in np.unique(N)))

for n in [60, 140, 300]:
    print(f"N = {n:3d}: the line predicts {b0 + b1 * n:4.2f} t/ha; this world's true mean is {true_curve(n):4.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Reading the output:

  • The slope. Each extra kilogram of nitrogen per hectare goes with 0.0202 t/ha more grain, which reads better as 20.2 kg of grain per kg of nitrogen: if a kilogram of nitrogen costs more than about 20 kg of maize, extra fertiliser does not pay at these rates. Notice the wording "goes with". Here the rates were randomised, so reading the slope as the effect of nitrogen (on average, on these plots, this season) is justified. With observational data, as the next lesson shows, a slope compares different units rather than describing what happens when you change one.
  • The intercept. 1.45 t/ha is the line's yield with no nitrogen, meaningful here because N=0N = 0 was in the trial. If the predictor were adult height in centimetres, the intercept would be the prediction for a person 0 cm tall. Interpret an intercept only when x=0x = 0 lies within the data; otherwise centre the predictor (x−xˉx - \bar x) so that the intercept becomes the prediction at the average.
  • Agreement. The formulas, np.linalg.lstsq and scipy.stats.linregress agree to the digits printed, and the residuals sum to zero, as the theorem promised.
  • The mean residual at each rate. It is negative at 0 (−0.54), positive in the middle (up to +0.34 at 60), and negative again at 120 and 140. The points bow above the line in the middle and fall below it at both ends: the true response is curved, rising steeply and then levelling off. Over this range the line is still a useful summary, because the curvature is modest compared with the plot-to-plot scatter. But the residuals have issued a warning about trusting the line outside the range, and the last three lines of output show why. We return to them below.
ExplanationH

How good is the fit? Residuals and R²

The variation of the yields around their mean, SST=∑(yi−yˉ)2\text{SST} = \sum (y_i - \bar y)^2, splits into two parts.

Theorem— Decomposition of variation

For a least-squares line with an intercept,

∑(yi−yˉ)2⏟SST (total)=∑(y^i−yˉ)2⏟SSR (explained by the line)+∑(yi−y^i)2⏟SSE (residual).\underbrace{\sum (y_i - \bar y)^2}_{\text{SST (total)}} = \underbrace{\sum (\hat y_i - \bar y)^2}_{\text{SSR (explained by the line)}} + \underbrace{\sum (y_i - \hat y_i)^2}_{\text{SSE (residual)}}.

Proof

Write yi−yˉ=(y^i−yˉ)+eiy_i - \bar y = (\hat y_i - \bar y) + e_i, square, and sum. The cross term is 2∑(y^i−yˉ)ei=2(b0−yˉ)∑ei+2b1∑xiei2\sum (\hat y_i - \bar y) e_i = 2(b_0 - \bar y) \sum e_i + 2 b_1 \sum x_i e_i, and the two normal equations make both sums zero. Geometrically this is Pythagoras' theorem: y^−yˉ1\hat{\mathbf y} - \bar y \mathbf 1 lies in the column space of XX, and e\mathbf e is perpendicular to it.

∎

The coefficient of determination R2=SSR/SST=1−SSE/SSTR^2 = \text{SSR}/\text{SST} = 1 - \text{SSE}/\text{SST} is the fraction of the variation in yy accounted for by the line. In simple regression it equals r2r^2, the squared correlation. For the trial R2=0.888R^2 = 0.888: the line accounts for about 89% of the plot-to-plot variation in yield, and the remaining 11% is scatter around the line.

R² is useful and much abused. Here is what it does not tell you.

  • Whether the line is the right shape. Our data are curved and R² is 0.888. In 1973 Francis Anscombe published four small datasets with identical means, variances, correlation, fitted line and R²: a straight-line cloud, a smooth curve, a perfect line spoiled by one outlier, and a vertical stack of points whose whole slope comes from one far-off point. Only a picture tells them apart.
  • How precise predictions are. R² depends on how spread out the xx values happen to be: the same relationship measured over rates 60 to 80 only would give a much smaller R², with the same scatter around the line. For prediction, the residual standard error s=SSE/(n−2)s = \sqrt{\text{SSE}/(n-2)}, in the units of yy, is the honest summary of how far typical points fall from the line.
  • Whether the relationship matters, or is causal. A treatment that lowers blood pressure by 5 mmHg on average may explain only a few per cent of the variation between patients and still prevent many strokes; two quantities that both grow over time can have a high R² and no causal link.

The working statistician's first check is a residual plot: residuals against fitted values, or against xx. A structureless horizontal band is what the model expects; a curve means the mean is not linear (our bow); a funnel means the variance is not constant; a few points far from the band are cases to investigate, not to delete quietly. The next lesson makes these checks systematic, with leverage and influence.

ComputationC

How sure are we about the slope?

Run the trial again next season on other plots and the slope would come out a little different. The slope is a statistic with a sampling distribution, exactly like the sample mean in estimation and confidence intervals.

Theorem— Sampling distribution of the slope

Under the model, b1b_1 is unbiased for β1\beta_1 and Var⁡(b1)=σ2/Sxx\operatorname{Var}(b_1) = \sigma^2 / S_{xx}. If the errors are normal, then with s2=SSE/(n−2)s^2 = \text{SSE}/(n-2),

T=b1−β1s/Sxx∼tn−2.T = \frac{b_1 - \beta_1}{s/\sqrt{S_{xx}}} \sim t_{n-2}.

Proof

Because ∑(xi−xˉ)=0\sum (x_i - \bar x) = 0, we can write b1=∑ciYib_1 = \sum c_i Y_i with fixed weights ci=(xi−xˉ)/Sxxc_i = (x_i - \bar x)/S_{xx}. These weights satisfy ∑ci=0\sum c_i = 0 and ∑cixi=1\sum c_i x_i = 1, so E[b1]=∑ci(β0+β1xi)=β1E[b_1] = \sum c_i(\beta_0 + \beta_1 x_i) = \beta_1. By independence, Var⁡(b1)=σ2∑ci2=σ2/Sxx\operatorname{Var}(b_1) = \sigma^2 \sum c_i^2 = \sigma^2 / S_{xx}, and a linear combination of independent normal variables is normal. The remaining step, that (n−2)s2/σ2(n-2)s^2/\sigma^2 has a χn−22\chi^2_{n-2} distribution independent of b1b_1, comes from the projection picture: the residual vector lies in the (n−2)(n-2)-dimensional subspace perpendicular to the columns of XX, and for normal errors projections onto perpendicular subspaces are independent. Dividing the standardised normal by the square root of the scaled chi-squared gives the tt distribution (see statistical inference in Lumenary Math).

∎

The formula SE(b1)=σ/Sxx\text{SE}(b_1) = \sigma/\sqrt{S_{xx}} carries a lesson for design: the slope is pinned down better when the xx values are more spread out, and when there are more of them. To measure a slope, measure across a wide range. That is why the trial used rates from 0 to 140 rather than from 60 to 80.

From the theorem come the standard tools: a 95% confidence interval b1±t0.975, n−2 SE(b1)b_1 \pm t_{0.975,\,n-2}\,\text{SE}(b_1), and a test of H0 ⁣:β1=0H_0\colon \beta_1 = 0 with statistic t=b1/SE(b1)t = b_1/\text{SE}(b_1), as in hypothesis testing. For predictions at a new value x0x_0 there are two different intervals, and confusing them is a classic error:

  • a confidence interval for the mean response, with standard error s1/n+(x0−xˉ)2/Sxxs\sqrt{1/n + (x_0 - \bar x)^2/S_{xx}}, says where the average yield of many plots given x0x_0 lies;
  • a prediction interval for one new observation, with standard error s1+1/n+(x0−xˉ)2/Sxxs\sqrt{1 + 1/n + (x_0 - \bar x)^2/S_{xx}}, says where a single new plot's yield is likely to fall. The extra 1 is the new plot's own scatter, and no amount of data makes it go away.
python
import numpy as np
from scipy import stats

rng = np.random.default_rng(2026)
N = np.repeat([0, 20, 40, 60, 80, 100, 120, 140], 3).astype(float)
yield_t = 1.2 + 4.0 * (1 - np.exp(-N / 110)) + rng.normal(0, 0.35, N.size)

n = N.size
Sxx = np.sum((N - N.mean()) ** 2)
b1 = np.sum((N - N.mean()) * (yield_t - yield_t.mean())) / Sxx
b0 = yield_t.mean() - b1 * N.mean()
s = np.sqrt(np.sum((yield_t - b0 - b1 * N) ** 2) / (n - 2))   # residual standard error
se_b1 = s / np.sqrt(Sxx)
t_crit = stats.t.ppf(0.975, df=n - 2)
t_stat = b1 / se_b1
p = 2 * stats.t.sf(abs(t_stat), df=n - 2)
print(f"s = {s:.3f} t/ha   SE(b1) = {se_b1:.5f}   t = {t_stat:.1f} on {n - 2} df   p = {p:.1e}")
print(f"95% CI for the slope: {b1 - t_crit * se_b1:.4f} to {b1 + t_crit * se_b1:.4f}")

for x0 in [100, 300]:
    mean0 = b0 + b1 * x0
    se_mean = s * np.sqrt(1 / n + (x0 - N.mean()) ** 2 / Sxx)
    se_new = s * np.sqrt(1 + 1 / n + (x0 - N.mean()) ** 2 / Sxx)
    print(f"N = {x0}: mean yield {mean0:.2f}, 95% CI {mean0 - t_crit * se_mean:.2f} to {mean0 + t_crit * se_mean:.2f}; "
          f"95% prediction interval for one new plot {mean0 - t_crit * se_new:.2f} to {mean0 + t_crit * se_new:.2f}")

# Does the formula tell the truth? Pretend the fitted line is the true one and rerun the trial 10,000 times.
sims = 10_000
Y = b0 + b1 * N + rng.normal(0, s, (sims, n))              # each row is one imagined trial
slopes = (Y - Y.mean(axis=1, keepdims=True)) @ (N - N.mean()) / Sxx
intercepts = Y.mean(axis=1) - slopes * N.mean()
s_sim = np.sqrt(np.sum((Y - intercepts[:, None] - slopes[:, None] * N) ** 2, axis=1) / (n - 2))
covered = np.abs(slopes - b1) <= t_crit * s_sim / np.sqrt(Sxx)
print(f"sd of the 10,000 slopes: {slopes.std(ddof=1):.5f}   formula sigma/sqrt(Sxx): {s / np.sqrt(Sxx):.5f}")
print(f"95% intervals that caught the true slope: {covered.mean():.1%}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The slope is 0.0202 with a standard error of 0.00153, so the 95% confidence interval runs from 0.0170 to 0.0233 t/ha per kg/ha: roughly 17 to 23 kg of grain per kg of nitrogen. The tt statistic of 13.2, with its tiny pp-value, says only that the slope is not zero, which no agronomist doubted; the interval is the informative part. At N=100N = 100 the mean yield is pinned down to within about 0.18 t/ha either way, yet a single new plot could plausibly yield anything from 2.74 to 4.20 t/ha.

History & global context

Galton, heights and regression to the mean

The word regression comes from a puzzle. In the 1880s the English scientist Francis Galton collected the heights of more than 900 adult children and their parents, from about 200 families. Children of very tall parents were, on average, tall, but less tall than their parents; children of very short parents were short, but less short. In an 1886 paper he called this "regression towards mediocrity", and the name stuck to the whole method, even though the phenomenon has nothing specially to do with heredity. (Galton's family data are in the HistData package for R. Galton also founded eugenics, a movement built on false and harmful ideas about human worth. Statistics owes him some of its tools, and owes it to its students to say so plainly.)

The explanation is in the least-squares formula. Measure both variables in standard units (subtract the mean, divide by the standard deviation). Then b1=r sy/sxb_1 = r\,s_y/s_x becomes simply rr, and the regression line reads

z^y=r zx.\hat z_y = r\, z_x.

Unless the correlation is perfect, ∣r∣<1|r| < 1, so the predicted value is always closer to the mean, in standard units, than the value it was predicted from. That is all regression to the mean is: a consequence of imperfect correlation, with no cause required. Whenever two measurements are imperfectly correlated (a test and a retest, this year's crop and next year's, a parent's height and a child's), the extreme cases on one will, on average, be less extreme on the other.

Here is a world with no cause at all. Each synthetic pupil has a fixed skill, and sits two tests a week apart; each score is skill plus luck on the day, and nothing happens between the tests.

python
import numpy as np

# 10,000 synthetic pupils sit two tests a week apart. Nothing happens in between.
rng = np.random.default_rng(7)
skill = rng.normal(60, 10, 10_000)              # what each pupil really knows
test1 = skill + rng.normal(0, 8, skill.size)    # plus luck on the day
test2 = skill + rng.normal(0, 8, skill.size)    # fresh luck, same skill

r = np.corrcoef(test1, test2)[0, 1]
slope = np.polyfit(test1, test2, 1)[0]
print(f"correlation r = {r:.3f}; slope of test2 on test1 = {slope:.3f}; theory 100/164 = {100 / 164:.3f}")
print(f"spread: sd(test1) = {test1.std():.2f}, sd(test2) = {test2.std():.2f}")

top = test1 >= np.quantile(test1, 0.9)
bottom = test1 <= np.quantile(test1, 0.1)
print(f"top 10% on test 1:    mean {test1[top].mean():.1f} -> {test2[top].mean():.1f} on test 2")
print(f"bottom 10% on test 1: mean {test1[bottom].mean():.1f} -> {test2[bottom].mean():.1f} on test 2")

top2 = test2 >= np.quantile(test2, 0.9)
print(f"top 10% on test 2:    mean {test2[top2].mean():.1f} on test 2, but {test1[top2].mean():.1f} on test 1")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The top tenth on test 1 averaged 82.1 and then 73.6; the bottom tenth rose from 37.7 to 46.5. The correlation is 0.603, close to the theoretical Var⁡(skill)/Var⁡(score)=100/164\operatorname{Var}(\text{skill})/\operatorname{Var}(\text{score}) = 100/164. Two further lines of output dispose of common misreadings. The spread does not shrink (standard deviations 12.64 and 12.83), so the pupils are not all drifting towards average. And the effect runs backwards in time too: the top tenth on test 2 averaged only 73.6 on test 1.

2020404060608080100100score on test 1score on test 2test 2 = test 1 (no regression)regression line, slope r = 0.61pupils scoring 82 on test 1 ...... average 73 on test 2where about 95% of pupils fall
Regression to the mean in the synthetic test scores. The shaded ellipse is where about 95% of pupils fall. If test 2 simply repeated test 1, points would scatter around the dashed line. The least-squares line is flatter, with slope r = 0.61, so pupils who scored 82 on test 1 average only about 73 on test 2.

Regression to the mean is behind many false conclusions. Speed cameras tend to be installed where collisions have recently spiked, and collisions at such sites would fall on average even without a camera, so careful evaluations estimate how much of the fall is regression. Patients enter clinical trials when their symptoms are worst, and improve on average even on placebo. Pupils given extra tutoring because they did worst in a mock exam improve, partly because of the tutoring and partly by regression; only a comparison with similar pupils who were not tutored, ideally chosen at random, can separate the two. That is the subject of experimental design.

ExplanationH

Beyond the data: extrapolation

Return to the last lines of the first program. At 300 kg/ha of nitrogen, about twice the largest rate in the trial, the line predicts 7.50 t/ha, and the prediction interval from the second program is 6.47 to 8.53 t/ha. In the synthetic world that generated the data, the true mean yield at 300 kg/ha is 4.94 t/ha, far outside the interval. Maize cannot use unlimited nitrogen: the response levels off, and the line, which fitted well enough between 0 and 140, sails on upwards. The data contained one hint of this, the bowed residuals, and nothing more.

The interval was not miscalculated. Every interval in this lesson is conditional on the model: it measures the uncertainty from noise, assuming the straight line holds. Its width grows only slowly with distance from xˉ\bar x, through the (x0−xˉ)2/Sxx(x_0 - \bar x)^2/S_{xx} term, and says nothing about the real danger outside the data, that the model's shape is wrong there. No statistical method can reveal the shape of a relationship where there are no data; only subject knowledge or new data can.

Extrapolation takes many forms:

  • Beyond the range of a predictor, as here. A related warning sign is a prediction that is physically impossible, such as a negative weight or a probability above 1.
  • Forward in time. A straight trend fitted to ten years of data and projected forty years ahead is a statement of "if the trend continues", not a forecast. Time series need their own methods, in time series.
  • To other populations. A trial on the sandy soils of one district says less about clay soils elsewhere; a drug trial in adults says less about children. Whether a result carries over is called external validity or transportability, and it is taken up in causal inference.

A grave example combines extrapolation with a second mistake. Before the launch of the space shuttle Challenger in January 1986, engineers and managers discussed whether cold damaged the rubber O-ring seals in the boosters. The forecast, around freezing, was far colder than any previous launch (the coldest had been about 12 °C), and part of the discussion looked only at flights on which seals had been damaged, among which the link with temperature looked weak. Including the undamaged flights shows damage concentrated at the lowest temperatures. The shuttle broke apart 73 seconds after launch, and seven crew members died. Statisticians still teach both lessons from it: never discard the cases where nothing happened, and treat predictions outside the data with the deepest suspicion.

ExplanationH

The professional view

In practice no one codes the formulas by hand. In R the tool is lm(); in Python, statsmodels gives the same table of estimates, standard errors, tt statistics and intervals. Both are shown for reference and are not run here:

r
# R (not run here)
trial <- data.frame(N = N, yield = yield_t)
fit <- lm(yield ~ N, data = trial)
summary(fit)                      # estimates, standard errors, t, p, R^2
confint(fit)                      # confidence intervals for the coefficients
predict(fit, data.frame(N = 100), interval = "prediction")
python
# Python with statsmodels (not run here)
import statsmodels.formula.api as smf
fit = smf.ols("yield_t ~ N", data=df).fit()
print(fit.summary())

scikit-learn's LinearRegression computes the same coefficients but deliberately reports no standard errors: it is built for prediction, where the question is how well the model predicts new data, not how precisely each coefficient is known. Neither view is wrong; they answer different questions.

A few things professionals know that textbooks sometimes leave out:

  • Numerics. Libraries never compute (X⊤X)−1(X^\top X)^{-1} explicitly, because doing so squares the condition number of the problem and can lose half the available digits. R's lm uses a QR decomposition of XX; NumPy's lstsq uses the singular value decomposition. See Lumenary Math's numerical methods.
  • Unequal variance. When residuals fan out, the coefficients are still unbiased, but the usual standard errors are wrong. Econometricians routinely report heteroscedasticity-consistent ("sandwich" or "robust") standard errors, which remain valid without the equal-variance assumption; transformations, in the next lesson, are another remedy.
  • Errors in the predictor. Least squares assumes xx is measured exactly. If xx carries measurement error (blood pressure measured once, self-reported diet), the slope is biased towards zero, an effect epidemiologists call regression dilution. Errors-in-variables methods correct it when the size of the measurement error is known.
Common misconceptionsH

Common misconceptions

Misconception

"Regressing yy on xx and xx on yy give the same line." They do not. The slope of yy on xx is r sy/sxr\,s_y/s_x; the slope of xx on yy is r sx/syr\,s_x/s_y. Drawn on the same axes, the two lines coincide only when ∣r∣=1|r| = 1. They answer different questions: predicting yy from xx minimises vertical misses, predicting xx from yy horizontal ones. Each line shows regression to the mean in its own direction.

Misconception

"Regression to the mean means that everything becomes average over time." It does not. In the simulation the spread of test 2 was the same as the spread of test 1. Individuals move in both directions; it is only the groups selected for being extreme that move, on average, towards the middle, because their selection favoured luck that does not repeat.

Misconception

"A tiny pp-value for the slope means a strong or important relationship." A pp-value measures how surprising the data would be if the slope were exactly zero. With enough data, a slope too small to matter is "highly significant". Report the slope with its confidence interval, in units someone can act on.

Misconception

"Linear regression needs yy to be normally distributed." The model assumes the errors around the line are normal, and even that matters mainly for small samples. The overall distribution of yy can be anything, since it reflects the spread of the xx values too. A histogram of yy is not a check of the model; a plot of residuals is.

Exercises

Exercises

1. Linear regression

Warm-upHuman

A survey of 500 households in Lima fits a regression of monthly household income (in soles) on the number of years of schooling of the head of the household. The slope is 120 soles per year, and R² is 0.18. Which statement is the best reading of the slope?

Choose one

2. Linear regression

CoreHuman

In a study of young trees in a Ugandan nursery, the age xx (months) and height yy (cm) have means xˉ=20\bar x = 20 and yˉ=70\bar y = 70, standard deviations sx=4s_x = 4 and sy=15s_y = 15, and correlation r=0.8r = 0.8. Using the least-squares line of yy on xx, what height (in cm) is predicted for a tree aged 25 months?

3. Linear regression

CoreHuman

A reading test is scored so that the mean is 100 and the standard deviation 15, both the first time pupils take it and when they retake it a month later. Scores on the two sittings have correlation r=0.5r = 0.5, and nothing in particular happens between them. Mei scored 130 the first time. What is the least-squares prediction of her retest score?

4. Linear regression

Warm-upHuman

A simple linear regression of the time for a 5 km run (minutes) on hours of training per week, for a club in Nairobi, has R2=0.49R^2 = 0.49 and a negative slope. What is the correlation between training hours and run time?

Choose one

5. Linear regression

CoreHuman

Chimwemwe's farmer plans to put 100 kg/ha of nitrogen on one field next season and asks: between what yields will my field probably fall? From the regression in the lesson, which interval should Chimwemwe give her?

Choose one

6. Linear regression

StretchHuman

Amara regresses daughters' heights yy on mothers' heights xx in a large sample where both have the same standard deviation, 6 cm, and the correlation is r=0.5r = 0.5. The slope is 0.5. She then regresses mothers' heights on daughters' heights. What slope does she get?

Choose one

7. Linear regression

CoreComputational

Write fit_line(x, y) that returns the least-squares intercept, slope and R2R^2 as a tuple (b0, b1, r2), using the formulas from the lesson: b1=Sxy/Sxxb_1 = S_{xy}/S_{xx}, b0=yˉ−b1xˉb_0 = \bar y - b_1 \bar x and R2=1−SSE/SSTR^2 = 1 - \text{SSE}/\text{SST}. The inputs may be lists or NumPy arrays.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Linear regression

StretchComputational

Write slope_ci(x, y, level=0.95) that returns the confidence interval (lo, hi) for the slope of a simple linear regression, using b1±t∗ s/Sxxb_1 \pm t^{*}\,s/\sqrt{S_{xx}} with s2=SSE/(n−2)s^2 = \text{SSE}/(n-2) and t∗t^{*} the appropriate quantile of the tt distribution with n−2n - 2 degrees of freedom (scipy.stats.t.ppf). It must work for any level, such as 0.90 or 0.99.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources

  • NIST/SEMATECH e-Handbook of Statistical Methods · National Institute of Standards and TechnologyFree. Its chapter on process modelling covers least squares, residual analysis and prediction intervals with engineering examples.
  • OpenIntro Statistics · David Diez, Mine Çetinkaya-Rundel and Christopher BarrA free, openly licensed textbook; its chapter on linear regression covers the same ground at a gentler pace.
  • R documentation: lm (fitting linear models)
  • statsmodels: linear regressionPython's library for regression with standard errors, tests and diagnostics.
  • scikit-learn: linear modelsLeast squares and its regularised relatives, from the prediction point of view.
  • HistData: data sets from the history of statistics (R package)Includes Galton's family height data, so you can redo his analysis.
  • Regression towards mediocrity in hereditary stature · Francis Galton · 1886Journal of the Anthropological Institute of Great Britain and Ireland, volume 15. The paper that named regression.
  • Graphs in statistical analysis · F. J. Anscombe · 1973The American Statistician, volume 27. Four datasets with identical regression output and very different pictures.