Skip to content
LumenaryAcademy
Modelling 75 min

Multiple regression and diagnostics

Several predictors at once, what "holding the others fixed" really means, categorical predictors and interactions, collinearity, the residual checks that keep a model honest, and Simpson's paradox.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Further out, but dearer?

Ana analyses housing for a city council in Porto. She has the monthly rents of 300 flats, with each flat's floor area, its distance from the city centre, its district and whether it has a balcony. (The data in this lesson are synthetic, generated by a program, with the structure such data really have.) A councillor asks a simple question: how much cheaper is it to live further out?

Ana fits the regression of the previous lesson, rent on distance, and gets a surprise: the slope is positive. Flats further from the centre rent for slightly more. Has she found a city where the suburbs are expensive? No. Flats further out are also larger, and larger flats cost more. Distance and area are tangled together, and a regression on distance alone credits distance with what area did. To answer the councillor's real question, which is about two flats that differ in distance but are otherwise alike, Ana needs to compare flats at the same area.

That is what multiple regression does. It fits several predictors at once and reports, for each one, how the response changes with that predictor when the others are held fixed. It is the workhorse of empirical science, from epidemiology to economics, and also the place where statistics demands the most judgement. What does "held fixed" mean when you never actually held anything? Which predictors belong in the model? And how do we know the model fits? The second half of this lesson is about diagnostics: the checks that tell us when a model is misleading us.

DefinitionsH

The model in matrix form

With p−1p - 1 predictors, the model for observation ii is

Yi=β0+β1xi1+β2xi2+⋯+βp−1xi,p−1+εi.Y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_{p-1} x_{i,p-1} + \varepsilon_i.

Stacking the nn observations, with a column of ones for the intercept, gives the compact form we met in linear regression.

Definition— linear model

Y=Xβ+ε\mathbf Y = X\boldsymbol\beta + \boldsymbol\varepsilon, where XX is the n×pn \times p design matrix (known, of full column rank pp), β\boldsymbol\beta the vector of pp unknown coefficients, and the errors satisfy E[ε]=0E[\boldsymbol\varepsilon] = \mathbf 0 and Var⁡(ε)=σ2I\operatorname{Var}(\boldsymbol\varepsilon) = \sigma^2 I: uncorrelated, with equal variance.

The least-squares estimate is again the orthogonal projection of y\mathbf y onto the column space of XX, now a pp-dimensional subspace of Rn\mathbb{R}^n:

b=(X⊤X)−1X⊤y,Var⁡(b)=σ2(X⊤X)−1,s2=SSEn−p.\mathbf b = (X^\top X)^{-1} X^\top \mathbf y, \qquad \operatorname{Var}(\mathbf b) = \sigma^2 (X^\top X)^{-1}, \qquad s^2 = \frac{\text{SSE}}{n - p}.

The variance formula follows because b=AY\mathbf b = A\mathbf Y with A=(X⊤X)−1X⊤A = (X^\top X)^{-1}X^\top, and Var⁡(AY)=A(σ2I)A⊤=σ2(X⊤X)−1\operatorname{Var}(A\mathbf Y) = A(\sigma^2 I)A^\top = \sigma^2 (X^\top X)^{-1}. Standard errors are the square roots of the diagonal of s2(X⊤X)−1s^2 (X^\top X)^{-1}, and the tt tests and confidence intervals work exactly as before, with n−pn - p degrees of freedom. The Gauss–Markov theorem adds that, under these assumptions, least squares has the smallest variance among all unbiased estimators that are linear in Y\mathbf Y; normality is not needed for that, only for exact tt and FF distributions in small samples.

"Full column rank" is not a technicality: if one column is an exact linear combination of others (area in square metres and area in square feet, say), X⊤XX^\top X cannot be inverted and the coefficients are not determined. The linear algebra is in Lumenary Math's linear algebra.

IntuitionH

What “holding the others fixed” means

Ana never held area fixed: she observed 300 flats as they were. So in what sense does the coefficient of distance describe a comparison "at the same area"? The precise answer is a theorem.

Theorem— Frisch–Waugh–Lovell

In the least-squares fit of y\mathbf y on the columns of X=[ X1    xj ]X = [\,X_1 \;\; \mathbf x_j\,], the coefficient of xj\mathbf x_j equals the slope of the simple regression of y~\tilde{\mathbf y} on x~j\tilde{\mathbf x}_j, where y~\tilde{\mathbf y} and x~j\tilde{\mathbf x}_j are the residuals from regressing y\mathbf y and xj\mathbf x_j, respectively, on the other columns X1X_1 (including the intercept).

Proof

Let M=I−X1(X1⊤X1)−1X1⊤M = I - X_1(X_1^\top X_1)^{-1}X_1^\top, the projection onto the orthogonal complement of the columns of X1X_1; then y~=My\tilde{\mathbf y} = M\mathbf y and x~j=Mxj\tilde{\mathbf x}_j = M\mathbf x_j. Write the full fit as y=X1b1+bjxj+e\mathbf y = X_1\mathbf b_1 + b_j\mathbf x_j + \mathbf e, where e\mathbf e is orthogonal to every column of XX. Apply MM: it annihilates X1b1X_1\mathbf b_1 and leaves e\mathbf e unchanged (since e\mathbf e is already orthogonal to X1X_1), so y~=bjx~j+e\tilde{\mathbf y} = b_j\tilde{\mathbf x}_j + \mathbf e. Because e\mathbf e is orthogonal to xj\mathbf x_j and to X1X_1, it is orthogonal to x~j=xj−(a combination of X1)\tilde{\mathbf x}_j = \mathbf x_j - (\text{a combination of } X_1). So e\mathbf e is the residual of a least-squares fit of y~\tilde{\mathbf y} on x~j\tilde{\mathbf x}_j (no intercept is needed, since both have mean zero), and its slope is bjb_j.

∎

In words: the multiple-regression coefficient of distance uses only the part of distance that area cannot predict, the variation in distance among flats of similar size, and relates it to the part of rent that area cannot predict. That is the statistical meaning of "holding area fixed". It is a statement about the data and the model, not about any experiment, and it inherits two limits: it holds area fixed only linearly (as the model describes area's effect), and it can only adjust for predictors that are in the model.

The theorem also explains why coefficients change when predictors are added. If distance and area were uncorrelated, residualising distance on area would change nothing, and the coefficient of distance would be the same with or without area. They are correlated, so it changes, and in Ana's data it changes sign.

Rents, like most prices and incomes, vary by percentages: a balcony adds a few per cent to a rent rather than a fixed amount, and the spread of rents grows with their level. So Ana models the logarithm of rent; the section on transformations below shows the evidence for that choice. A coefficient bb on the log scale means that a one-unit increase in the predictor multiplies the rent by ebe^{b}, a change of 100(eb−1)100(e^{b} - 1) per cent, which for small bb is close to 100b100b per cent.

python
import numpy as np
import pandas as pd

# 300 synthetic flats in one city. Bigger flats tend to be further from the centre.
rng = np.random.default_rng(42)
n = 300
dist = rng.uniform(0.5, 15, n)                                   # km from the centre
area = np.clip(40 + 5 * dist + rng.normal(0, 12, n), 25, None)   # floor area, m^2
district = rng.choice(["North", "Riverside", "East"], n)
balcony = (rng.random(n) < 0.4).astype(float)
log_rent = (6.0 + 0.009 * area - 0.035 * dist + 0.05 * balcony
            + np.select([district == "Riverside", district == "East"], [0.10, -0.08], 0.0)
            + 0.003 * (area - 70) * (district == "Riverside")
            + rng.normal(0, 0.12, n))
flats = pd.DataFrame({"rent": np.exp(log_rent).round().astype(int), "area": area.round(1), "dist": dist.round(2),
                      "district": district, "balcony": balcony.astype(int)})

print(flats.head(4).to_string(index=False))

def ols(X, y):
    """Least squares with an intercept; returns coefficients and their standard errors."""
    X = np.column_stack([np.ones(len(y)), X])
    b = np.linalg.lstsq(X, y, rcond=None)[0]
    resid = y - X @ b
    s2 = resid @ resid / (len(y) - X.shape[1])
    return b, np.sqrt(np.diag(s2 * np.linalg.inv(X.T @ X)))

y = np.log(flats["rent"].to_numpy())
d, a = flats["dist"].to_numpy(), flats["area"].to_numpy()
print(f"\ncorrelation of distance and area: {np.corrcoef(d, a)[0, 1]:.2f}")
b, se = ols(d, y)
print(f"log rent ~ distance:         distance {b[1]:+.4f} (SE {se[1]:.4f})")
b, se = ols(np.column_stack([d, a]), y)
print(f"log rent ~ distance + area:  distance {b[1]:+.4f} (SE {se[1]:.4f}), area {b[2]:+.4f} (SE {se[2]:.4f})")
print(f"one more km at the same area: {100 * np.expm1(b[1]):+.1f}% rent; 10 more m^2 at the same distance: {100 * np.expm1(10 * b[2]):+.1f}%")

# Frisch-Waugh-Lovell: remove from distance (and from log rent) what area can explain, then regress.
d_left = d - np.polyval(np.polyfit(a, d, 1), a)
y_left = y - np.polyval(np.polyfit(a, y, 1), a)
print(f"slope of what is left of log rent on what is left of distance: {np.polyfit(d_left, y_left, 1)[0]:+.4f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

On its own, distance has a slope of +0.0114 on the log scale: about 1.1% more rent per km. With area in the model, the coefficient of distance becomes −0.0383, so that among flats of the same area each extra kilometre goes with 3.8% less rent, and each extra 10 m² at the same distance with 10.7% more. The last line confirms the theorem: regressing what is left of log rent on what is left of distance, after removing what area explains from both, gives exactly −0.0383. The correlation of 0.86 between distance and area is what made the two answers differ.

ComputationC

Categorical predictors and interactions

District is not a number. To put a categorical predictor with kk levels into a regression, choose one level as the reference and add k−1k - 1 indicator (dummy) columns, each 1 for flats in that level and 0 otherwise. With North as the reference, the coefficient of Riverside is the difference in log rent between a Riverside flat and an otherwise identical North flat. Adding a kk-th column for North as well would make the indicators sum to the intercept column, and XX would lose full rank, a mistake known as the dummy variable trap. A binary predictor such as "has a balcony" is the special case k=2k = 2: one column.

So far each predictor has had one coefficient, the same everywhere: the model says an extra square metre is worth the same percentage in every district. An interaction relaxes this. Adding the product column area × Riverside lets the slope of area differ in Riverside: the area slope is βarea\beta_{\text{area}} in North and East and βarea+βarea×Riverside\beta_{\text{area}} + \beta_{\text{area} \times \text{Riverside}} in Riverside. Centring area at 70 m² before forming the product keeps the main effects interpretable: the Riverside coefficient is then the district difference for a 70 m² flat, rather than for a flat of zero area.

To ask whether district matters at all, we need to test three coefficients together (Riverside, East and the interaction). The F test for nested models compares the residual sum of squares with and without the terms:

F=(SSEsmall−SSEfull)/qSSEfull/(n−p),F = \frac{(\text{SSE}_{\text{small}} - \text{SSE}_{\text{full}})/q}{\text{SSE}_{\text{full}}/(n - p)},

which has an Fq, n−pF_{q,\,n-p} distribution if the qq extra coefficients are all zero. With one group factor and nothing else, this is exactly the one-way analysis of variance of experimental design and ANOVA.

python
import numpy as np
import pandas as pd

# 300 synthetic flats in one city. Bigger flats tend to be further from the centre.
rng = np.random.default_rng(42)
n = 300
dist = rng.uniform(0.5, 15, n)                                   # km from the centre
area = np.clip(40 + 5 * dist + rng.normal(0, 12, n), 25, None)   # floor area, m^2
district = rng.choice(["North", "Riverside", "East"], n)
balcony = (rng.random(n) < 0.4).astype(float)
log_rent = (6.0 + 0.009 * area - 0.035 * dist + 0.05 * balcony
            + np.select([district == "Riverside", district == "East"], [0.10, -0.08], 0.0)
            + 0.003 * (area - 70) * (district == "Riverside")
            + rng.normal(0, 0.12, n))
flats = pd.DataFrame({"rent": np.exp(log_rent).round().astype(int), "area": area.round(1), "dist": dist.round(2),
                      "district": district, "balcony": balcony.astype(int)})

from scipy import stats

y = np.log(flats["rent"].to_numpy())
area_c = flats["area"].to_numpy() - 70                  # centred: coefficients refer to a 70 m^2 flat
river = (flats["district"] == "Riverside").to_numpy().astype(float)
east = (flats["district"] == "East").to_numpy().astype(float)
cols = {"intercept": np.ones(len(y)), "area-70": area_c, "dist": flats["dist"].to_numpy(),
        "balcony": flats["balcony"].to_numpy().astype(float), "Riverside": river, "East": east,
        "area x Riverside": area_c * river}

def fit(names):
    X = np.column_stack([cols[c] for c in names])
    b = np.linalg.lstsq(X, y, rcond=None)[0]
    sse = np.sum((y - X @ b) ** 2)
    se = np.sqrt(np.diag(sse / (len(y) - len(names)) * np.linalg.inv(X.T @ X)))
    return b, se, sse

print("first three rows of the design matrix:")
print(pd.DataFrame({c: cols[c][:3] for c in cols}).round(2).to_string(index=False))

full = list(cols)
b, se, sse_full = fit(full)
print(f"\n{'term':>17} {'estimate':>9} {'SE':>7} {'t':>6}   as % change")
for name, est, s in zip(full, b, se):
    pct = "" if name == "intercept" else f"{100 * np.expm1(est):+6.1f}%" + (" per m^2" if "area" in name else " per km" if name == "dist" else "")
    print(f"{name:>17} {est:9.4f} {s:7.4f} {est / s:6.1f}   {pct}")

# Does district matter at all? Compare nested models with an F test.
no_district = ["intercept", "area-70", "dist", "balcony"]
_, _, sse_small = fit(no_district)
q, df_resid = len(full) - len(no_district), len(y) - len(full)
F = ((sse_small - sse_full) / q) / (sse_full / df_resid)
print(f"\nF test for district (3 terms): F = {F:.1f} on ({q}, {df_resid}) df, p = {stats.f.sf(F, q, df_resid):.1e}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The design matrix shows how categories become numbers: the first flat is in East, so its East column is 1; the second is in Riverside, so its Riverside column is 1 and its interaction column copies its centred area, 3.1. Reading the table: a balcony goes with 4.8% more rent, other things equal; a 70 m² Riverside flat rents for 11.7% more than a similar North flat, and an East flat for 5.8% less. The interaction is clear: each square metre adds about 1.0% in North and East, and about 0.0098+0.0045=0.01430.0098 + 0.0045 = 0.0143 on the log scale, roughly 1.4%, in Riverside, where larger flats (perhaps with views of the river) command a growing premium. The F test says district as a whole matters overwhelmingly (F=61.2F = 61.2 on 3 and 293 degrees of freedom).

ExplanationH

Collinearity

Suppose Ana also has the number of rooms in each flat. Rooms and area say nearly the same thing: correlation 0.90 in her data. Putting both in the model is not wrong, but it makes the model ask the data a hard question: among flats with the same area, how does the number of rooms matter, and among flats with the same number of rooms, how does area matter? There is little variation of that kind, so the answers are imprecise.

The precise statement is a formula for the variance of one coefficient:

Var⁡(bj)=σ2(n−1) sxj2⋅11−Rj2,\operatorname{Var}(b_j) = \frac{\sigma^2}{(n-1)\, s_{x_j}^2} \cdot \frac{1}{1 - R_j^2},

where Rj2R_j^2 is the R2R^2 from regressing xjx_j on all the other predictors. The first factor is the variance in a simple regression; the second, VIFj=1/(1−Rj2)\text{VIF}_j = 1/(1 - R_j^2), is the variance inflation factor. It is the Frisch–Waugh–Lovell theorem again: only the fraction 1−Rj21 - R_j^2 of xjx_j's variation is left after the other predictors have taken their share. A VIF of 10 means the standard error is 10≈3.2\sqrt{10} \approx 3.2 times what it would be if xjx_j were uncorrelated with the others.

Collinearity is a shortage of information, not a violation of any assumption. The estimates remain unbiased, and predictions within the range of the data remain good: the model cannot tell area's contribution from rooms', but it knows their sum well. The remedies are therefore about the question: drop one of two near-duplicates if they measure the same thing, combine them into an index, collect data that break the correlation, or accept wide intervals honestly. Regularisation (ridge regression), which trades a little bias for much less variance, appears in high-dimensional statistics.

ComputationC

Residuals, leverage and influence

Residual plots, met in the previous lesson, remain the first check: residuals against fitted values, against each predictor, and against anything not in the model (time of data collection, the person who recorded it). Multiple regression adds two ideas, because a point can be unusual in its predictors as well as in its response.

Definition— hat matrix and leverage

The hat matrix H=X(X⊤X)−1X⊤H = X(X^\top X)^{-1}X^\top maps the responses to the fitted values, y^=Hy\hat{\mathbf y} = H\mathbf y. Its diagonal entry hiih_{ii} is the leverage of observation ii: how strongly yiy_i pulls its own fitted value. Leverages lie between 1/n1/n and 11 and sum to pp, so their average is p/np/n. A common flag is hii>2p/nh_{ii} > 2p/n.

Leverage depends only on the predictors: a flat far from the others in area and distance has high leverage whatever its rent. Since Var⁡(ei)=σ2(1−hii)\operatorname{Var}(e_i) = \sigma^2(1 - h_{ii}), residuals of high-leverage points are smaller than their errors, because the line bends towards them. Dividing by the right standard deviation gives the studentised residual ri=ei/(s1−hii)r_i = e_i / (s\sqrt{1 - h_{ii}}), comparable across points.

Definition— Cook's distance

Cook's distance measures how far all the fitted values move when observation ii is deleted. It can be computed without refitting:

Di=ri2p⋅hii1−hii.D_i = \frac{r_i^2}{p}\cdot\frac{h_{ii}}{1 - h_{ii}}.

Influence is the product of surprise (ri2r_i^2) and leverage. Values above about 1, or far above the rest, deserve a look.

outlier, low leverageslope 0.86 → 0.86high leverage, on the trendslope 0.86 → 0.82high leverage, off the trendslope 0.86 → 0.13
Three kinds of unusual point (orange), with the line fitted without it (dashed) and with it (blue). Left: an outlier in the response at an ordinary x has low leverage and leaves the slope unchanged. Middle: a point far out in x but on the trend has high leverage and little influence. Right: a point far out in x and off the trend has high leverage and high influence; it drags the slope from 0.86 to 0.13.

The program below checks collinearity with VIFs, then plants one data-entry error in Ana's data, a large central flat whose weekly rent was typed in as monthly, and lets the diagnostics find it. Finally it compares the residual spread of the raw-rent and log-rent models.

python
import numpy as np
import pandas as pd

# 300 synthetic flats in one city. Bigger flats tend to be further from the centre.
rng = np.random.default_rng(42)
n = 300
dist = rng.uniform(0.5, 15, n)                                   # km from the centre
area = np.clip(40 + 5 * dist + rng.normal(0, 12, n), 25, None)   # floor area, m^2
district = rng.choice(["North", "Riverside", "East"], n)
balcony = (rng.random(n) < 0.4).astype(float)
log_rent = (6.0 + 0.009 * area - 0.035 * dist + 0.05 * balcony
            + np.select([district == "Riverside", district == "East"], [0.10, -0.08], 0.0)
            + 0.003 * (area - 70) * (district == "Riverside")
            + rng.normal(0, 0.12, n))
flats = pd.DataFrame({"rent": np.exp(log_rent).round().astype(int), "area": area.round(1), "dist": dist.round(2),
                      "district": district, "balcony": balcony.astype(int)})


def design(df):
    return np.column_stack([np.ones(len(df)), df["area"], df["dist"], df["balcony"],
                            df["district"] == "Riverside", df["district"] == "East"]).astype(float)

# 1. Collinearity: add the number of rooms, which is almost a function of floor area.
rooms = np.clip(np.round(flats["area"] / 25 + rng.normal(0, 0.4, len(flats))), 1, None)
def vif(X, j):
    """1 / (1 - R^2) from regressing column j on the other columns (all include the intercept)."""
    others = np.delete(X, j, axis=1)
    fitted = others @ np.linalg.lstsq(others, X[:, j], rcond=None)[0]
    r2 = 1 - np.sum((X[:, j] - fitted) ** 2) / np.sum((X[:, j] - X[:, j].mean()) ** 2)
    return 1 / (1 - r2)
X = design(flats)
X_rooms = np.column_stack([X, rooms])
print(f"corr(area, rooms) = {np.corrcoef(flats['area'], rooms)[0, 1]:.2f}")
print(f"VIF: area {vif(X_rooms, 1):.1f}, distance {vif(X_rooms, 2):.1f}, rooms {vif(X_rooms, 6):.1f}")
y = np.log(flats["rent"].to_numpy())
for label, M in [("without rooms", X), ("with rooms", X_rooms)]:
    b = np.linalg.lstsq(M, y, rcond=None)[0]
    s2 = np.sum((y - M @ b) ** 2) / (len(y) - M.shape[1])
    print(f"{label:>13}: area coefficient {b[1]:.4f}, SE {np.sqrt(s2 * np.linalg.inv(M.T @ M)[1, 1]):.4f}")

# 2. Leverage and influence. Add one data-entry error: a large central flat whose weekly rent was typed as monthly.
bad = pd.DataFrame({"rent": [260], "area": [190.0], "dist": [1.2], "district": ["North"], "balcony": [1]})
data = pd.concat([flats, bad], ignore_index=True)
X, y = design(data), np.log(data["rent"].to_numpy())
n, p = X.shape
H = X @ np.linalg.inv(X.T @ X) @ X.T                     # the hat matrix: fitted = H @ y
h = np.diag(H)
b = np.linalg.lstsq(X, y, rcond=None)[0]
e = y - X @ b
s2 = e @ e / (n - p)
r_stud = e / np.sqrt(s2 * (1 - h))                       # internally studentised residuals
cooks = r_stud ** 2 * h / (p * (1 - h))
print(f"\nmean leverage p/n = {p / n:.3f}; trace of H = {h.sum():.1f}")
for i in np.argsort(cooks)[::-1][:3]:
    print(f"row {i}: area {data.area[i]:5.1f}, dist {data.dist[i]:5.2f}, rent {data.rent[i]:4d}, "
          f"leverage {h[i]:.3f}, studentised residual {r_stud[i]:+.1f}, Cook's D {cooks[i]:.2f}")
b_clean = np.linalg.lstsq(X[:-1], y[:-1], rcond=None)[0]
print(f"area coefficient with the bad row {b[1]:.4f}, without it {b_clean[1]:.4f}")

# 3. Why log rent? Spread of residuals in the cheapest and dearest thirds of fitted values.
X, rent = design(flats), flats["rent"].to_numpy().astype(float)
for label, resp in [("rent", rent), ("log rent", np.log(rent))]:
    fitted = X @ np.linalg.lstsq(X, resp, rcond=None)[0]
    res = resp - fitted
    lo, hi = np.quantile(fitted, [1 / 3, 2 / 3])
    ratio = res[fitted > hi].std() / res[fitted < lo].std()
    print(f"{label:>8}: residual sd in dearest third / cheapest third = {ratio:.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Finding an influential point is the beginning of an investigation, not a licence to delete it. Here the record is a typing error, so it should be corrected or removed, and the change documented. A genuine flat that is simply unusual (a penthouse) is information: it may show that the model's form fails for large flats, and the honest responses are to model that, or to state that conclusions apply only to the range of ordinary flats. Deleting inconvenient points until the result looks good is one of the "forking paths" that make published findings fail to replicate, a theme of multiple testing and reproducibility.

ExplanationH

Transformations

The last two lines of the program are the evidence for modelling log rent. With rent itself as the response, residuals in the dearest third of flats are 1.42 times as spread out as in the cheapest third: a funnel, the signature of variation that grows with the level. With log rent the ratio is 1.07, close to the constant spread the model assumes. Transforming the response repairs two things at once: it stabilises the variance, and it makes effects that are naturally multiplicative (percentages) additive.

A few transformations do most of the work in practice.

  • Log of the response for positive quantities that vary by percentages: prices, incomes, concentrations, counts of cells or species. Coefficients become percentage changes. If some values are zero, the log is not available, and a model for counts (the next lesson) is usually better than adding an arbitrary constant.
  • Log of a predictor when its effect shows diminishing returns: the difference between incomes of 10,000 and 20,000 matters more than between 110,000 and 120,000. With both sides logged, the coefficient is an elasticity: the percentage change in yy per 1% change in xx.
  • Polynomial and spline terms, such as adding x2x^2, for curvature like the nitrogen response of the previous lesson. The model is still linear in the coefficients, so everything in this lesson applies.
  • Box–Cox transformations y(λ)=(yλ−1)/λy^{(\lambda)} = (y^\lambda - 1)/\lambda form a family that includes the log (λ→0\lambda \to 0) and lets the data suggest a power. In practice, a value of λ\lambda that is interpretable (1, ½, 0, −1) is usually preferred over the exact optimum.

Two cautions. A model for log⁡y\log y predicts the mean of log⁡y\log y; exponentiating gives the median rent, not the mean, which is larger by a factor of about es2/2e^{s^2/2}. And a transformation changes the question: a test on the log scale is about ratios, not differences.

Worked examplesH

Simpson's paradox

Ana's positive-then-negative slope is an example of a general phenomenon. Here it is in a cleaner form, with synthetic measurements shaped like the openly licensed Palmer penguins data from Antarctica: bill length and bill depth for three species.

python
import numpy as np

# Synthetic bill measurements (mm) for three penguin species, shaped like the Palmer penguins data.
rng = np.random.default_rng(1)
species = {"Adelie": (150, 38.8, 18.3), "Gentoo": (120, 47.5, 15.0), "Chinstrap": (70, 48.8, 18.4)}
length, depth, group = [], [], []
for name, (n, mean_len, mean_dep) in species.items():
    L = rng.normal(mean_len, 2.8, n)
    length.append(L)
    depth.append(mean_dep + 0.22 * (L - mean_len) + rng.normal(0, 0.8, n))
    group += [name] * n
length, depth, group = np.concatenate(length), np.concatenate(depth), np.array(group)

pooled = np.polyfit(length, depth, 1)[0]
print(f"all birds together: slope {pooled:+.3f} mm of depth per mm of length, r = {np.corrcoef(length, depth)[0, 1]:+.2f}")
for name in species:
    m = group == name
    print(f"{name:>10} only:    slope {np.polyfit(length[m], depth[m], 1)[0]:+.3f}, r = {np.corrcoef(length[m], depth[m])[0, 1]:+.2f}")

# Multiple regression with species as a categorical predictor: one common slope, three intercepts.
X = np.column_stack([length, group == "Gentoo", group == "Chinstrap", np.ones(len(length))]).astype(float)
b = np.linalg.lstsq(X, depth, rcond=None)[0]
print(f"depth ~ length + species: slope {b[0]:+.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Within every species, birds with longer bills have deeper bills: slopes of +0.29, +0.21 and +0.27 mm per mm, with correlations around +0.6 to +0.7. Pool the three species and the relationship turns negative: −0.074, with r=−0.22r = -0.22. Adding species as a categorical predictor recovers a common within-species slope of +0.26. The figure shows why: Gentoo penguins have long but shallow bills, so the pooled cloud runs downhill from the Adélie group (short and deep) to the Gentoo group (long and shallow), although every group runs uphill.

303540455055121416182022bill length (mm)bill depth (mm)AdélieGentooChinstrap
Simpson's paradox in synthetic penguin bills (30 birds of each species shown). Within each species (solid blue lines), longer bills are deeper. Pooling the species (dashed orange line) reverses the sign of the relationship, because the species differ in both length and depth.

The reversal has a long history. Karl Pearson and George Udny Yule noticed it around 1900, and Edward Simpson analysed it for tables of counts in 1951. The best-known real case is the 1973 graduate admissions at the University of California, Berkeley, analysed by Bickel, Hammel and O'Connell in 1975: men were admitted at a noticeably higher overall rate than women, yet in most individual departments women were admitted at a similar or higher rate. Women had applied more often to departments that admitted few applicants of either sex. Medicine has its own examples, in which a treatment looks better overall but worse within every severity group, because doctors gave it more often to milder cases.

So should one always adjust for the grouping variable? No. Here, species is a cause of both bill length and bill depth, a confounder, and adjusting for it answers the biological question about bills within a species. But if a variable lies on the path from cause to effect (a mediator), adjusting for it removes part of the effect you want to measure; and if it is a common consequence of the two variables (a collider), adjusting for it can create an association where none exists. Deciding what to adjust for needs a causal model, usually drawn as a diagram, and that is the subject of causal inference.

confounder: adjustZXYmediator: do not adjustMXYcollider: do not adjustCXYX: exposure or treatment · Y: outcome · arrows: direct causal effects
Three causal roles a third variable can play, drawn as causal diagrams. Left: Z causes both X and Y (a confounder, like species for bill length and depth); adjusting for it removes a spurious part of the association. Middle: M lies on the path from X to Y (a mediator); adjusting for it removes part of the effect being estimated. Right: C is caused by both X and Y (a collider); adjusting for it, or selecting data on it, creates a spurious association.
ExplanationH

The professional view

Statisticians fit these models with R's lm() and formula syntax, or with Python's statsmodels, which builds dummy and interaction columns from a formula and provides leverage, Cook's distance and VIFs ready-made. Shown for reference, not run here:

r
# R (not run here)
flats$area_c <- flats$area - 70
flats$riverside <- flats$district == "Riverside"
fit <- lm(log(rent) ~ area_c + dist + balcony + district + area_c:riverside, data = flats)
summary(fit)
plot(fit)                       # residuals vs fitted, Q-Q plot, scale-location, leverage with Cook's contours
car::vif(fit)                   # variance inflation factors, from the car package

A few habits of experienced analysts:

  • Decide the comparison before the model. Choose predictors from subject knowledge and the question, ideally written down before seeing the results; selecting them by trying many models and keeping significant ones invalidates the standard errors. Model selection with honest accounting is in likelihood and model selection.
  • Look at the data at every step. Scatter-plot matrices before fitting; residual plots after. Many errors (a weekly rent typed as monthly, a missing-value code of 999) are visible in a plot long before they distort a coefficient.
  • Report effects on interpretable scales, with intervals: "3.8% less rent per km (95% CI …)" rather than a tt statistic.
  • Use robust standard errors when variance is unequal, and cluster-robust standard errors when observations come in groups (flats in the same building, pupils in the same school); grouped data are treated properly in multilevel models.
  • Remember the data's people. Housing, health and income data describe real lives. Use the least identifying data that answer the question, aggregate before publishing, and never release records that could identify a tenant.
Common misconceptionsH

Common misconceptions

Misconception

"A coefficient tells you what would happen if you changed that predictor." Only if the data come from an experiment, or if the model contains every confounder and no mediators or colliders. In observational data, a coefficient is a comparison between units that differ in that predictor and match on the others in the model, as the Frisch–Waugh–Lovell theorem makes precise.

Misconception

"If a predictor's coefficient is not significant, it does not matter." With collinearity, two predictors can each be non-significant while together they are highly significant: the data know their combined effect but cannot split it. Test them together with an F test, and look at the VIFs before concluding anything.

Misconception

"Outliers should be removed." An outlier is a question, not an error. Remove a point only for a documented reason (a recording mistake, a unit outside the population studied), and report results with and without influential points when they matter.

Misconception

"More predictors always give a better model." Adding predictors never lowers R2R^2 on the data used for fitting, but it can raise the variance of every coefficient, adjust away part of the effect of interest (a mediator), or introduce bias (a collider). Each predictor should earn its place for a reason connected to the question.

Exercises

Exercises

1. Multiple regression and diagnostics

Warm-upHuman

In Ana's model of log rent, the coefficient of the balcony indicator is 0.047, with area, distance and district also in the model. Which statement is the best reading?

Choose one

2. Multiple regression and diagnostics

CoreHuman

In a regression of house prices on several predictors, regressing the predictor floor area on all the other predictors gives R2=0.96R^2 = 0.96. By what factor is the standard error of the area coefficient multiplied, compared with what it would be if area were uncorrelated with the other predictors (all else equal)?

3. Multiple regression and diagnostics

CoreHuman

In the lesson's model of log rent, area is centred at 70 m², and the estimates include 0.00980.0098 for area − 70 and 0.00450.0045 for area × Riverside. What is the estimated change in log rent for one extra square metre in Riverside?

Choose one

4. Multiple regression and diagnostics

CoreHuman

A regression with an intercept and six predictors is fitted to 350 observations. What is the sum of the leverages h11+h22+⋯+h350,350h_{11} + h_{22} + \cdots + h_{350,350}?

5. Multiple regression and diagnostics

CoreHuman

Synthetic records from two hospitals give these survival rates after an operation:

Mild cases Severe cases All cases
Hospital A 800 of 850 (94.1%) 90 of 150 (60.0%) 890 of 1000 (89.0%)
Hospital B 190 of 200 (95.0%) 520 of 800 (65.0%) 710 of 1000 (71.0%)

A patient's case is known to be severe. Based on these data alone, which statement is best?

Choose one

6. Multiple regression and diagnostics

StretchHuman

A study asks whether a new fertiliser increases a farm's income. Fertiliser raises yield, and higher yield raises income. A researcher proposes a regression of income on fertiliser use and yield, "to control for yield". What is the main problem?

Choose one

7. Multiple regression and diagnostics

CoreComputational

Write influence(X, y) that returns two NumPy arrays: the leverages hiih_{ii} and Cook's distances DiD_i for a least-squares fit. X is the full design matrix, already including a column of ones. Use the hat matrix H=X(X⊤X)−1X⊤H = X(X^\top X)^{-1}X^\top, the studentised residuals ri=ei/(s1−hii)r_i = e_i/(s\sqrt{1 - h_{ii}}) with s2=SSE/(n−p)s^2 = \text{SSE}/(n - p), and Di=ri2hii/(p(1−hii))D_i = r_i^2 h_{ii}/(p(1 - h_{ii})). The tests check your Cook's distances against brute-force refits with each point deleted.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Multiple regression and diagnostics

StretchComputational

Write vif(X) that returns a NumPy array with the variance inflation factor of each predictor column of the design matrix X (column 0 is the intercept and gets no VIF). For predictor jj, regress column jj on all the other columns (including the intercept), compute that regression's Rj2R_j^2, and return 1/(1−Rj2)1/(1 - R_j^2).

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources

  • Regression and Other Stories · Andrew Gelman, Jennifer Hill and Aki Vehtari · 2020The book's website, with data and code for every example. Unusually good on interpreting coefficients, interactions and transformations.
  • NIST/SEMATECH e-Handbook of Statistical Methods · National Institute of Standards and TechnologyIts process-modelling chapter covers residual plots, transformations and model validation.
  • statsmodels: regression diagnostics and influence measures
  • palmerpenguins: Palmer Archipelago penguin data · Allison Horst, Alison Hill and Kristen GormanAn openly licensed (CC0) dataset of real penguin measurements in which bill length and depth show Simpson's paradox.
  • Sex bias in graduate admissions, data from Berkeley · P. J. Bickel, E. A. Hammel and J. W. O'Connell · 1975Science, volume 187. The best-known real example of Simpson's paradox in a table of counts.
  • An analysis of transformations · G. E. P. Box and D. R. Cox · 1964Journal of the Royal Statistical Society, Series B, volume 26. The Box–Cox family of power transformations.