Further out, but dearer?
Ana analyses housing for a city council in Porto. She has the monthly rents of 300 flats, with each flat's floor area, its distance from the city centre, its district and whether it has a balcony. (The data in this lesson are synthetic, generated by a program, with the structure such data really have.) A councillor asks a simple question: how much cheaper is it to live further out?
Ana fits the regression of the previous lesson, rent on distance, and gets a surprise: the slope is positive. Flats further from the centre rent for slightly more. Has she found a city where the suburbs are expensive? No. Flats further out are also larger, and larger flats cost more. Distance and area are tangled together, and a regression on distance alone credits distance with what area did. To answer the councillor's real question, which is about two flats that differ in distance but are otherwise alike, Ana needs to compare flats at the same area.
That is what multiple regression does. It fits several predictors at once and reports, for each one, how the response changes with that predictor when the others are held fixed. It is the workhorse of empirical science, from epidemiology to economics, and also the place where statistics demands the most judgement. What does "held fixed" mean when you never actually held anything? Which predictors belong in the model? And how do we know the model fits? The second half of this lesson is about diagnostics: the checks that tell us when a model is misleading us.
The model in matrix form
With predictors, the model for observation is
Stacking the observations, with a column of ones for the intercept, gives the compact form we met in linear regression.
Definition— linear model
, where is the design matrix (known, of full column rank ), the vector of unknown coefficients, and the errors satisfy and : uncorrelated, with equal variance.
The least-squares estimate is again the orthogonal projection of onto the column space of , now a -dimensional subspace of :
The variance formula follows because with , and . Standard errors are the square roots of the diagonal of , and the tests and confidence intervals work exactly as before, with degrees of freedom. The Gauss–Markov theorem adds that, under these assumptions, least squares has the smallest variance among all unbiased estimators that are linear in ; normality is not needed for that, only for exact and distributions in small samples.
"Full column rank" is not a technicality: if one column is an exact linear combination of others (area in square metres and area in square feet, say), cannot be inverted and the coefficients are not determined. The linear algebra is in Lumenary Math's linear algebra.
What “holding the others fixed” means
Ana never held area fixed: she observed 300 flats as they were. So in what sense does the coefficient of distance describe a comparison "at the same area"? The precise answer is a theorem.
Theorem— Frisch–Waugh–Lovell
In the least-squares fit of on the columns of , the coefficient of equals the slope of the simple regression of on , where and are the residuals from regressing and , respectively, on the other columns (including the intercept).
Proof
Let , the projection onto the orthogonal complement of the columns of ; then and . Write the full fit as , where is orthogonal to every column of . Apply : it annihilates and leaves unchanged (since is already orthogonal to ), so . Because is orthogonal to and to , it is orthogonal to . So is the residual of a least-squares fit of on (no intercept is needed, since both have mean zero), and its slope is .
∎
In words: the multiple-regression coefficient of distance uses only the part of distance that area cannot predict, the variation in distance among flats of similar size, and relates it to the part of rent that area cannot predict. That is the statistical meaning of "holding area fixed". It is a statement about the data and the model, not about any experiment, and it inherits two limits: it holds area fixed only linearly (as the model describes area's effect), and it can only adjust for predictors that are in the model.
The theorem also explains why coefficients change when predictors are added. If distance and area were uncorrelated, residualising distance on area would change nothing, and the coefficient of distance would be the same with or without area. They are correlated, so it changes, and in Ana's data it changes sign.
Rents, like most prices and incomes, vary by percentages: a balcony adds a few per cent to a rent rather than a fixed amount, and the spread of rents grows with their level. So Ana models the logarithm of rent; the section on transformations below shows the evidence for that choice. A coefficient on the log scale means that a one-unit increase in the predictor multiplies the rent by , a change of per cent, which for small is close to per cent.
import numpy as np
import pandas as pd
# 300 synthetic flats in one city. Bigger flats tend to be further from the centre.
rng = np.random.default_rng(42)
n = 300
dist = rng.uniform(0.5, 15, n) # km from the centre
area = np.clip(40 + 5 * dist + rng.normal(0, 12, n), 25, None) # floor area, m^2
district = rng.choice(["North", "Riverside", "East"], n)
balcony = (rng.random(n) < 0.4).astype(float)
log_rent = (6.0 + 0.009 * area - 0.035 * dist + 0.05 * balcony
+ np.select([district == "Riverside", district == "East"], [0.10, -0.08], 0.0)
+ 0.003 * (area - 70) * (district == "Riverside")
+ rng.normal(0, 0.12, n))
flats = pd.DataFrame({"rent": np.exp(log_rent).round().astype(int), "area": area.round(1), "dist": dist.round(2),
"district": district, "balcony": balcony.astype(int)})
print(flats.head(4).to_string(index=False))
def ols(X, y):
"""Least squares with an intercept; returns coefficients and their standard errors."""
X = np.column_stack([np.ones(len(y)), X])
b = np.linalg.lstsq(X, y, rcond=None)[0]
resid = y - X @ b
s2 = resid @ resid / (len(y) - X.shape[1])
return b, np.sqrt(np.diag(s2 * np.linalg.inv(X.T @ X)))
y = np.log(flats["rent"].to_numpy())
d, a = flats["dist"].to_numpy(), flats["area"].to_numpy()
print(f"\ncorrelation of distance and area: {np.corrcoef(d, a)[0, 1]:.2f}")
b, se = ols(d, y)
print(f"log rent ~ distance: distance {b[1]:+.4f} (SE {se[1]:.4f})")
b, se = ols(np.column_stack([d, a]), y)
print(f"log rent ~ distance + area: distance {b[1]:+.4f} (SE {se[1]:.4f}), area {b[2]:+.4f} (SE {se[2]:.4f})")
print(f"one more km at the same area: {100 * np.expm1(b[1]):+.1f}% rent; 10 more m^2 at the same distance: {100 * np.expm1(10 * b[2]):+.1f}%")
# Frisch-Waugh-Lovell: remove from distance (and from log rent) what area can explain, then regress.
d_left = d - np.polyval(np.polyfit(a, d, 1), a)
y_left = y - np.polyval(np.polyfit(a, y, 1), a)
print(f"slope of what is left of log rent on what is left of distance: {np.polyfit(d_left, y_left, 1)[0]:+.4f}")On its own, distance has a slope of +0.0114 on the log scale: about 1.1% more rent per km. With area in the model, the coefficient of distance becomes −0.0383, so that among flats of the same area each extra kilometre goes with 3.8% less rent, and each extra 10 m² at the same distance with 10.7% more. The last line confirms the theorem: regressing what is left of log rent on what is left of distance, after removing what area explains from both, gives exactly −0.0383. The correlation of 0.86 between distance and area is what made the two answers differ.
Categorical predictors and interactions
District is not a number. To put a categorical predictor with levels into a regression, choose one level as the reference and add indicator (dummy) columns, each 1 for flats in that level and 0 otherwise. With North as the reference, the coefficient of Riverside is the difference in log rent between a Riverside flat and an otherwise identical North flat. Adding a -th column for North as well would make the indicators sum to the intercept column, and would lose full rank, a mistake known as the dummy variable trap. A binary predictor such as "has a balcony" is the special case : one column.
So far each predictor has had one coefficient, the same everywhere: the model says an extra square metre is worth the same percentage in every district. An interaction relaxes this. Adding the product column area × Riverside lets the slope of area differ in Riverside: the area slope is in North and East and in Riverside. Centring area at 70 m² before forming the product keeps the main effects interpretable: the Riverside coefficient is then the district difference for a 70 m² flat, rather than for a flat of zero area.
To ask whether district matters at all, we need to test three coefficients together (Riverside, East and the interaction). The F test for nested models compares the residual sum of squares with and without the terms:
which has an distribution if the extra coefficients are all zero. With one group factor and nothing else, this is exactly the one-way analysis of variance of experimental design and ANOVA.
import numpy as np
import pandas as pd
# 300 synthetic flats in one city. Bigger flats tend to be further from the centre.
rng = np.random.default_rng(42)
n = 300
dist = rng.uniform(0.5, 15, n) # km from the centre
area = np.clip(40 + 5 * dist + rng.normal(0, 12, n), 25, None) # floor area, m^2
district = rng.choice(["North", "Riverside", "East"], n)
balcony = (rng.random(n) < 0.4).astype(float)
log_rent = (6.0 + 0.009 * area - 0.035 * dist + 0.05 * balcony
+ np.select([district == "Riverside", district == "East"], [0.10, -0.08], 0.0)
+ 0.003 * (area - 70) * (district == "Riverside")
+ rng.normal(0, 0.12, n))
flats = pd.DataFrame({"rent": np.exp(log_rent).round().astype(int), "area": area.round(1), "dist": dist.round(2),
"district": district, "balcony": balcony.astype(int)})
from scipy import stats
y = np.log(flats["rent"].to_numpy())
area_c = flats["area"].to_numpy() - 70 # centred: coefficients refer to a 70 m^2 flat
river = (flats["district"] == "Riverside").to_numpy().astype(float)
east = (flats["district"] == "East").to_numpy().astype(float)
cols = {"intercept": np.ones(len(y)), "area-70": area_c, "dist": flats["dist"].to_numpy(),
"balcony": flats["balcony"].to_numpy().astype(float), "Riverside": river, "East": east,
"area x Riverside": area_c * river}
def fit(names):
X = np.column_stack([cols[c] for c in names])
b = np.linalg.lstsq(X, y, rcond=None)[0]
sse = np.sum((y - X @ b) ** 2)
se = np.sqrt(np.diag(sse / (len(y) - len(names)) * np.linalg.inv(X.T @ X)))
return b, se, sse
print("first three rows of the design matrix:")
print(pd.DataFrame({c: cols[c][:3] for c in cols}).round(2).to_string(index=False))
full = list(cols)
b, se, sse_full = fit(full)
print(f"\n{'term':>17} {'estimate':>9} {'SE':>7} {'t':>6} as % change")
for name, est, s in zip(full, b, se):
pct = "" if name == "intercept" else f"{100 * np.expm1(est):+6.1f}%" + (" per m^2" if "area" in name else " per km" if name == "dist" else "")
print(f"{name:>17} {est:9.4f} {s:7.4f} {est / s:6.1f} {pct}")
# Does district matter at all? Compare nested models with an F test.
no_district = ["intercept", "area-70", "dist", "balcony"]
_, _, sse_small = fit(no_district)
q, df_resid = len(full) - len(no_district), len(y) - len(full)
F = ((sse_small - sse_full) / q) / (sse_full / df_resid)
print(f"\nF test for district (3 terms): F = {F:.1f} on ({q}, {df_resid}) df, p = {stats.f.sf(F, q, df_resid):.1e}")The design matrix shows how categories become numbers: the first flat is in East, so its East column is 1; the second is in Riverside, so its Riverside column is 1 and its interaction column copies its centred area, 3.1. Reading the table: a balcony goes with 4.8% more rent, other things equal; a 70 m² Riverside flat rents for 11.7% more than a similar North flat, and an East flat for 5.8% less. The interaction is clear: each square metre adds about 1.0% in North and East, and about on the log scale, roughly 1.4%, in Riverside, where larger flats (perhaps with views of the river) command a growing premium. The F test says district as a whole matters overwhelmingly ( on 3 and 293 degrees of freedom).
Collinearity
Suppose Ana also has the number of rooms in each flat. Rooms and area say nearly the same thing: correlation 0.90 in her data. Putting both in the model is not wrong, but it makes the model ask the data a hard question: among flats with the same area, how does the number of rooms matter, and among flats with the same number of rooms, how does area matter? There is little variation of that kind, so the answers are imprecise.
The precise statement is a formula for the variance of one coefficient:
where is the from regressing on all the other predictors. The first factor is the variance in a simple regression; the second, , is the variance inflation factor. It is the Frisch–Waugh–Lovell theorem again: only the fraction of 's variation is left after the other predictors have taken their share. A VIF of 10 means the standard error is times what it would be if were uncorrelated with the others.
Collinearity is a shortage of information, not a violation of any assumption. The estimates remain unbiased, and predictions within the range of the data remain good: the model cannot tell area's contribution from rooms', but it knows their sum well. The remedies are therefore about the question: drop one of two near-duplicates if they measure the same thing, combine them into an index, collect data that break the correlation, or accept wide intervals honestly. Regularisation (ridge regression), which trades a little bias for much less variance, appears in high-dimensional statistics.
Residuals, leverage and influence
Residual plots, met in the previous lesson, remain the first check: residuals against fitted values, against each predictor, and against anything not in the model (time of data collection, the person who recorded it). Multiple regression adds two ideas, because a point can be unusual in its predictors as well as in its response.
Definition— hat matrix and leverage
The hat matrix maps the responses to the fitted values, . Its diagonal entry is the leverage of observation : how strongly pulls its own fitted value. Leverages lie between and and sum to , so their average is . A common flag is .
Leverage depends only on the predictors: a flat far from the others in area and distance has high leverage whatever its rent. Since , residuals of high-leverage points are smaller than their errors, because the line bends towards them. Dividing by the right standard deviation gives the studentised residual , comparable across points.
Definition— Cook's distance
Cook's distance measures how far all the fitted values move when observation is deleted. It can be computed without refitting:
Influence is the product of surprise () and leverage. Values above about 1, or far above the rest, deserve a look.
The program below checks collinearity with VIFs, then plants one data-entry error in Ana's data, a large central flat whose weekly rent was typed in as monthly, and lets the diagnostics find it. Finally it compares the residual spread of the raw-rent and log-rent models.
import numpy as np
import pandas as pd
# 300 synthetic flats in one city. Bigger flats tend to be further from the centre.
rng = np.random.default_rng(42)
n = 300
dist = rng.uniform(0.5, 15, n) # km from the centre
area = np.clip(40 + 5 * dist + rng.normal(0, 12, n), 25, None) # floor area, m^2
district = rng.choice(["North", "Riverside", "East"], n)
balcony = (rng.random(n) < 0.4).astype(float)
log_rent = (6.0 + 0.009 * area - 0.035 * dist + 0.05 * balcony
+ np.select([district == "Riverside", district == "East"], [0.10, -0.08], 0.0)
+ 0.003 * (area - 70) * (district == "Riverside")
+ rng.normal(0, 0.12, n))
flats = pd.DataFrame({"rent": np.exp(log_rent).round().astype(int), "area": area.round(1), "dist": dist.round(2),
"district": district, "balcony": balcony.astype(int)})
def design(df):
return np.column_stack([np.ones(len(df)), df["area"], df["dist"], df["balcony"],
df["district"] == "Riverside", df["district"] == "East"]).astype(float)
# 1. Collinearity: add the number of rooms, which is almost a function of floor area.
rooms = np.clip(np.round(flats["area"] / 25 + rng.normal(0, 0.4, len(flats))), 1, None)
def vif(X, j):
"""1 / (1 - R^2) from regressing column j on the other columns (all include the intercept)."""
others = np.delete(X, j, axis=1)
fitted = others @ np.linalg.lstsq(others, X[:, j], rcond=None)[0]
r2 = 1 - np.sum((X[:, j] - fitted) ** 2) / np.sum((X[:, j] - X[:, j].mean()) ** 2)
return 1 / (1 - r2)
X = design(flats)
X_rooms = np.column_stack([X, rooms])
print(f"corr(area, rooms) = {np.corrcoef(flats['area'], rooms)[0, 1]:.2f}")
print(f"VIF: area {vif(X_rooms, 1):.1f}, distance {vif(X_rooms, 2):.1f}, rooms {vif(X_rooms, 6):.1f}")
y = np.log(flats["rent"].to_numpy())
for label, M in [("without rooms", X), ("with rooms", X_rooms)]:
b = np.linalg.lstsq(M, y, rcond=None)[0]
s2 = np.sum((y - M @ b) ** 2) / (len(y) - M.shape[1])
print(f"{label:>13}: area coefficient {b[1]:.4f}, SE {np.sqrt(s2 * np.linalg.inv(M.T @ M)[1, 1]):.4f}")
# 2. Leverage and influence. Add one data-entry error: a large central flat whose weekly rent was typed as monthly.
bad = pd.DataFrame({"rent": [260], "area": [190.0], "dist": [1.2], "district": ["North"], "balcony": [1]})
data = pd.concat([flats, bad], ignore_index=True)
X, y = design(data), np.log(data["rent"].to_numpy())
n, p = X.shape
H = X @ np.linalg.inv(X.T @ X) @ X.T # the hat matrix: fitted = H @ y
h = np.diag(H)
b = np.linalg.lstsq(X, y, rcond=None)[0]
e = y - X @ b
s2 = e @ e / (n - p)
r_stud = e / np.sqrt(s2 * (1 - h)) # internally studentised residuals
cooks = r_stud ** 2 * h / (p * (1 - h))
print(f"\nmean leverage p/n = {p / n:.3f}; trace of H = {h.sum():.1f}")
for i in np.argsort(cooks)[::-1][:3]:
print(f"row {i}: area {data.area[i]:5.1f}, dist {data.dist[i]:5.2f}, rent {data.rent[i]:4d}, "
f"leverage {h[i]:.3f}, studentised residual {r_stud[i]:+.1f}, Cook's D {cooks[i]:.2f}")
b_clean = np.linalg.lstsq(X[:-1], y[:-1], rcond=None)[0]
print(f"area coefficient with the bad row {b[1]:.4f}, without it {b_clean[1]:.4f}")
# 3. Why log rent? Spread of residuals in the cheapest and dearest thirds of fitted values.
X, rent = design(flats), flats["rent"].to_numpy().astype(float)
for label, resp in [("rent", rent), ("log rent", np.log(rent))]:
fitted = X @ np.linalg.lstsq(X, resp, rcond=None)[0]
res = resp - fitted
lo, hi = np.quantile(fitted, [1 / 3, 2 / 3])
ratio = res[fitted > hi].std() / res[fitted < lo].std()
print(f"{label:>8}: residual sd in dearest third / cheapest third = {ratio:.2f}")Finding an influential point is the beginning of an investigation, not a licence to delete it. Here the record is a typing error, so it should be corrected or removed, and the change documented. A genuine flat that is simply unusual (a penthouse) is information: it may show that the model's form fails for large flats, and the honest responses are to model that, or to state that conclusions apply only to the range of ordinary flats. Deleting inconvenient points until the result looks good is one of the "forking paths" that make published findings fail to replicate, a theme of multiple testing and reproducibility.
Transformations
The last two lines of the program are the evidence for modelling log rent. With rent itself as the response, residuals in the dearest third of flats are 1.42 times as spread out as in the cheapest third: a funnel, the signature of variation that grows with the level. With log rent the ratio is 1.07, close to the constant spread the model assumes. Transforming the response repairs two things at once: it stabilises the variance, and it makes effects that are naturally multiplicative (percentages) additive.
A few transformations do most of the work in practice.
- Log of the response for positive quantities that vary by percentages: prices, incomes, concentrations, counts of cells or species. Coefficients become percentage changes. If some values are zero, the log is not available, and a model for counts (the next lesson) is usually better than adding an arbitrary constant.
- Log of a predictor when its effect shows diminishing returns: the difference between incomes of 10,000 and 20,000 matters more than between 110,000 and 120,000. With both sides logged, the coefficient is an elasticity: the percentage change in per 1% change in .
- Polynomial and spline terms, such as adding , for curvature like the nitrogen response of the previous lesson. The model is still linear in the coefficients, so everything in this lesson applies.
- Box–Cox transformations form a family that includes the log () and lets the data suggest a power. In practice, a value of that is interpretable (1, ½, 0, −1) is usually preferred over the exact optimum.
Two cautions. A model for predicts the mean of ; exponentiating gives the median rent, not the mean, which is larger by a factor of about . And a transformation changes the question: a test on the log scale is about ratios, not differences.
Simpson's paradox
Ana's positive-then-negative slope is an example of a general phenomenon. Here it is in a cleaner form, with synthetic measurements shaped like the openly licensed Palmer penguins data from Antarctica: bill length and bill depth for three species.
import numpy as np
# Synthetic bill measurements (mm) for three penguin species, shaped like the Palmer penguins data.
rng = np.random.default_rng(1)
species = {"Adelie": (150, 38.8, 18.3), "Gentoo": (120, 47.5, 15.0), "Chinstrap": (70, 48.8, 18.4)}
length, depth, group = [], [], []
for name, (n, mean_len, mean_dep) in species.items():
L = rng.normal(mean_len, 2.8, n)
length.append(L)
depth.append(mean_dep + 0.22 * (L - mean_len) + rng.normal(0, 0.8, n))
group += [name] * n
length, depth, group = np.concatenate(length), np.concatenate(depth), np.array(group)
pooled = np.polyfit(length, depth, 1)[0]
print(f"all birds together: slope {pooled:+.3f} mm of depth per mm of length, r = {np.corrcoef(length, depth)[0, 1]:+.2f}")
for name in species:
m = group == name
print(f"{name:>10} only: slope {np.polyfit(length[m], depth[m], 1)[0]:+.3f}, r = {np.corrcoef(length[m], depth[m])[0, 1]:+.2f}")
# Multiple regression with species as a categorical predictor: one common slope, three intercepts.
X = np.column_stack([length, group == "Gentoo", group == "Chinstrap", np.ones(len(length))]).astype(float)
b = np.linalg.lstsq(X, depth, rcond=None)[0]
print(f"depth ~ length + species: slope {b[0]:+.3f}")Within every species, birds with longer bills have deeper bills: slopes of +0.29, +0.21 and +0.27 mm per mm, with correlations around +0.6 to +0.7. Pool the three species and the relationship turns negative: −0.074, with . Adding species as a categorical predictor recovers a common within-species slope of +0.26. The figure shows why: Gentoo penguins have long but shallow bills, so the pooled cloud runs downhill from the Adélie group (short and deep) to the Gentoo group (long and shallow), although every group runs uphill.
The reversal has a long history. Karl Pearson and George Udny Yule noticed it around 1900, and Edward Simpson analysed it for tables of counts in 1951. The best-known real case is the 1973 graduate admissions at the University of California, Berkeley, analysed by Bickel, Hammel and O'Connell in 1975: men were admitted at a noticeably higher overall rate than women, yet in most individual departments women were admitted at a similar or higher rate. Women had applied more often to departments that admitted few applicants of either sex. Medicine has its own examples, in which a treatment looks better overall but worse within every severity group, because doctors gave it more often to milder cases.
So should one always adjust for the grouping variable? No. Here, species is a cause of both bill length and bill depth, a confounder, and adjusting for it answers the biological question about bills within a species. But if a variable lies on the path from cause to effect (a mediator), adjusting for it removes part of the effect you want to measure; and if it is a common consequence of the two variables (a collider), adjusting for it can create an association where none exists. Deciding what to adjust for needs a causal model, usually drawn as a diagram, and that is the subject of causal inference.
The professional view
Statisticians fit these models with R's lm() and formula syntax, or with Python's statsmodels, which builds dummy and interaction columns from a formula and provides leverage, Cook's distance and VIFs ready-made. Shown for reference, not run here:
# R (not run here)
flats$area_c <- flats$area - 70
flats$riverside <- flats$district == "Riverside"
fit <- lm(log(rent) ~ area_c + dist + balcony + district + area_c:riverside, data = flats)
summary(fit)
plot(fit) # residuals vs fitted, Q-Q plot, scale-location, leverage with Cook's contours
car::vif(fit) # variance inflation factors, from the car packageA few habits of experienced analysts:
- Decide the comparison before the model. Choose predictors from subject knowledge and the question, ideally written down before seeing the results; selecting them by trying many models and keeping significant ones invalidates the standard errors. Model selection with honest accounting is in likelihood and model selection.
- Look at the data at every step. Scatter-plot matrices before fitting; residual plots after. Many errors (a weekly rent typed as monthly, a missing-value code of 999) are visible in a plot long before they distort a coefficient.
- Report effects on interpretable scales, with intervals: "3.8% less rent per km (95% CI …)" rather than a statistic.
- Use robust standard errors when variance is unequal, and cluster-robust standard errors when observations come in groups (flats in the same building, pupils in the same school); grouped data are treated properly in multilevel models.
- Remember the data's people. Housing, health and income data describe real lives. Use the least identifying data that answer the question, aggregate before publishing, and never release records that could identify a tenant.
Common misconceptions
Misconception
"A coefficient tells you what would happen if you changed that predictor." Only if the data come from an experiment, or if the model contains every confounder and no mediators or colliders. In observational data, a coefficient is a comparison between units that differ in that predictor and match on the others in the model, as the Frisch–Waugh–Lovell theorem makes precise.
Misconception
"If a predictor's coefficient is not significant, it does not matter." With collinearity, two predictors can each be non-significant while together they are highly significant: the data know their combined effect but cannot split it. Test them together with an F test, and look at the VIFs before concluding anything.
Misconception
"Outliers should be removed." An outlier is a question, not an error. Remove a point only for a documented reason (a recording mistake, a unit outside the population studied), and report results with and without influential points when they matter.
Misconception
"More predictors always give a better model." Adding predictors never lowers on the data used for fitting, but it can raise the variance of every coefficient, adjust away part of the effect of interest (a mediator), or introduce bias (a collider). Each predictor should earn its place for a reason connected to the question.
Exercises
1. Multiple regression and diagnostics
Warm-upHumanIn Ana's model of log rent, the coefficient of the balcony indicator is 0.047, with area, distance and district also in the model. Which statement is the best reading?
2. Multiple regression and diagnostics
CoreHumanIn a regression of house prices on several predictors, regressing the predictor floor area on all the other predictors gives . By what factor is the standard error of the area coefficient multiplied, compared with what it would be if area were uncorrelated with the other predictors (all else equal)?
3. Multiple regression and diagnostics
CoreHumanIn the lesson's model of log rent, area is centred at 70 m², and the estimates include for area − 70 and for area × Riverside. What is the estimated change in log rent for one extra square metre in Riverside?
4. Multiple regression and diagnostics
CoreHumanA regression with an intercept and six predictors is fitted to 350 observations. What is the sum of the leverages ?
5. Multiple regression and diagnostics
CoreHumanSynthetic records from two hospitals give these survival rates after an operation:
| Mild cases | Severe cases | All cases | |
|---|---|---|---|
| Hospital A | 800 of 850 (94.1%) | 90 of 150 (60.0%) | 890 of 1000 (89.0%) |
| Hospital B | 190 of 200 (95.0%) | 520 of 800 (65.0%) | 710 of 1000 (71.0%) |
A patient's case is known to be severe. Based on these data alone, which statement is best?
6. Multiple regression and diagnostics
StretchHumanA study asks whether a new fertiliser increases a farm's income. Fertiliser raises yield, and higher yield raises income. A researcher proposes a regression of income on fertiliser use and yield, "to control for yield". What is the main problem?
7. Multiple regression and diagnostics
CoreComputationalWrite influence(X, y) that returns two NumPy arrays: the leverages and Cook's distances for a least-squares fit. X is the full design matrix, already including a column of ones. Use the hat matrix , the studentised residuals with , and . The tests check your Cook's distances against brute-force refits with each point deleted.
8. Multiple regression and diagnostics
StretchComputationalWrite vif(X) that returns a NumPy array with the variance inflation factor of each predictor column of the design matrix X (column 0 is the intercept and gets no VIF). For predictor , regress column on all the other columns (including the intercept), compute that regression's , and return .
Further reading and resources
- Regression and Other Stories · Andrew Gelman, Jennifer Hill and Aki Vehtari · 2020The book's website, with data and code for every example. Unusually good on interpreting coefficients, interactions and transformations.
- NIST/SEMATECH e-Handbook of Statistical Methods · National Institute of Standards and TechnologyIts process-modelling chapter covers residual plots, transformations and model validation.
- statsmodels: regression diagnostics and influence measures
- palmerpenguins: Palmer Archipelago penguin data · Allison Horst, Alison Hill and Kristen GormanAn openly licensed (CC0) dataset of real penguin measurements in which bill length and depth show Simpson's paradox.
- Sex bias in graduate admissions, data from Berkeley · P. J. Bickel, E. A. Hammel and J. W. O'Connell · 1975Science, volume 187. The best-known real example of Simpson's paradox in a table of counts.
- An analysis of transformations · G. E. P. Box and D. R. Cox · 1964Journal of the Royal Statistical Society, Series B, volume 26. The Box–Cox family of power transformations.