When the response is yes, no, or how many
Wanjiru is an ecologist surveying 200 ponds in the highlands of Kenya for a species of frog. For each pond she records one thing about the frogs, present or absent, and two things about the pond: its area, and whether predatory fish live in it. A few hundred kilometres away, a road-safety engineer in Nairobi is studying 80 junctions: for each, how many collisions happened last year, how much traffic passed through, and whether the junction is a roundabout. (Both datasets in this lesson are synthetic, generated by programs that mimic real surveys.)
Both want what regression offers: how does the response change with the predictors, and how sure are we? But neither response is a continuous measurement with constant scatter around a line.
- A yes-or-no outcome has a probability, which must stay between 0 and 1. A straight line will eventually leave that range; fitted to Wanjiru's data it predicts a "probability" of −0.25 for small ponds with fish, as we shall see.
- A count cannot be negative, and its variance grows with its mean: a junction that averages 2 collisions a year varies by a collision or two, one that averages 20 by four or five. Effects are naturally multiplicative too: a roundabout might cut collisions by a third, whatever the traffic.
Generalised linear models (GLMs) keep everything good about linear regression (a linear combination of predictors, coefficients with standard errors, tests of nested models) and change two things: the distribution of the response, and the scale on which the predictors act. John Nelder and Robert Wedderburn, working at Rothamsted Experimental Station in England, unified these models in 1972, showing that one fitting algorithm serves them all. Logistic regression is now the standard tool for binary outcomes in medicine, ecology and the social sciences, and Poisson regression for counts in epidemiology, insurance and traffic safety.
Three ingredients
Definition— generalised linear model
A GLM has three parts:
- a random component: given the predictors, has a distribution from the exponential family (normal, binomial, Poisson, gamma, …) with mean ;
- a linear predictor: , or ;
- a link function , smooth and monotone, connecting them: , so .
The link stretches the restricted range of the mean onto the whole real line, where a linear predictor can roam freely. The common choices:
| Response | Distribution | Variance | Usual link | Model name |
|---|---|---|---|---|
| Measurement | Normal | identity, | linear regression | |
| Yes/no, or of | Binomial | per trial | logit, | logistic regression |
| Count | Poisson | log, | Poisson regression | |
| Positive, skewed (costs, durations) | Gamma | log | gamma regression |
The distributions themselves are described in distributions and, with their mathematics, in Lumenary Math's probability theory. Each distribution has a canonical link that makes the mathematics especially tidy (logit for the binomial, log for the Poisson, identity for the normal), but other links are legitimate. For binary data the probit link, the inverse of the standard normal distribution function, gives almost the same fits as the logit and dates back to bioassay in the 1930s; the complementary log-log link suits probabilities that approach 1 asymmetrically. Linear regression is the GLM with a normal distribution and the identity link, so everything here generalises the previous two lessons.
For a probability , the odds are : a probability of 0.8 is odds of 4 (to 1). The logit is the log of the odds. In logistic regression
so increasing by one unit adds to the log-odds, which multiplies the odds by , the odds ratio. In Poisson regression with the log link, a unit increase in multiplies the expected count by , the rate ratio.
Fitting by maximum likelihood and Newton's method
Least squares no longer fits the bill: the variance changes from observation to observation, and the mean is a curve. GLMs are fitted by maximum likelihood, choosing the coefficients that make the observed data most probable (the general theory is in likelihood and model selection). For logistic regression with , the probability of the data is , and its logarithm, after a line of algebra, is
Theorem— Score, information and Newton's method for logistic regression
The gradient (score) and Hessian of the logistic log-likelihood are
The Hessian is negative definite when has full column rank, so is strictly concave and has at most one maximum. Newton's method updates
and at the maximum, estimates the covariance matrix of .
Proof
Since , the derivative of the -th term with respect to is ; summing gives . Differentiating again, , so the Hessian is . For any , , because every weight is positive and by full rank. Newton's method replaces by its second-order Taylor expansion at the current and jumps to that quadratic's maximum, which is the update shown. The covariance statement is the general large-sample result that the inverse of the Fisher information, here , is the asymptotic covariance of a maximum-likelihood estimator.
∎
The update can be rewritten as a weighted least-squares fit of a "working response" with weights , so the algorithm is also called iteratively reweighted least squares (IRLS). It is how R's glm and statsmodels fit every GLM: for any canonical link, the score is and the information is with the variance function evaluated at , so one short loop fits them all. Newton's method converges quadratically near the maximum, roughly doubling the number of correct digits each step; the numerical analysis is in Lumenary Math's numerical methods.
Logistic regression for the frog survey
The program generates Wanjiru's synthetic survey, fits the logistic regression by Newton's method from scratch, and checks the coefficients with scikit-learn, an independent implementation that uses a different optimiser. Pond area enters as of the area in square metres, centred at 2.5 (about 300 m²), so one unit means a pond ten times larger.
import numpy as np
from scipy import stats
from sklearn.linear_model import LogisticRegression
# Synthetic survey of 200 ponds: is a frog species present (1) or not (0)?
rng = np.random.default_rng(12)
n = 200
log_area = rng.normal(2.5, 0.6, n) # log10 of pond area in m^2 (roughly 30 to 3000 m^2)
fish = (rng.random(n) < 0.45).astype(float) # predatory fish present?
eta = -0.5 + 2.0 * (log_area - 2.5) - 1.5 * fish # the true log-odds in this synthetic world
frog = (rng.random(n) < 1 / (1 + np.exp(-eta))).astype(float)
X = np.column_stack([np.ones(n), log_area - 2.5, fish])
def fit_logistic(X, y, verbose=False):
"""Newton's method on the logistic log-likelihood. Returns coefficients, covariance, log-likelihood."""
beta = np.zeros(X.shape[1])
for step in range(50):
p = 1 / (1 + np.exp(-X @ beta))
loglik = np.sum(y * np.log(p) + (1 - y) * np.log(1 - p))
if verbose and step < 5:
print(f"step {step}: beta = {np.round(beta, 4)}, log-likelihood {loglik:.4f}")
score = X.T @ (y - p) # gradient of the log-likelihood
info = X.T @ (X * (p * (1 - p))[:, None]) # minus the Hessian: X^T W X
delta = np.linalg.solve(info, score)
beta = beta + delta
if np.max(np.abs(delta)) < 1e-10:
break
p = 1 / (1 + np.exp(-X @ beta))
loglik = np.sum(y * np.log(p) + (1 - y) * np.log(1 - p))
return beta, np.linalg.inv(X.T @ (X * (p * (1 - p))[:, None])), loglik
beta, cov, ll = fit_logistic(X, frog, verbose=True)
se = np.sqrt(np.diag(cov))
sk = LogisticRegression(penalty=None, tol=1e-10, max_iter=1000).fit(X[:, 1:], frog)
print("Newton: ", np.round(beta, 4))
print("scikit-learn:", np.round(np.r_[sk.intercept_, sk.coef_[0]], 4))
for name, b, s in zip(["log10 area", "fish"], beta[1:], se[1:]):
print(f"{name:>10}: coefficient {b:+.3f} (SE {s:.3f}), odds ratio {np.exp(b):.2f}, "
f"95% CI {np.exp(b - 1.96 * s):.2f} to {np.exp(b + 1.96 * s):.2f}")
# Deviances and a likelihood-ratio test for fish
ll_null = fit_logistic(X[:, :1], frog)[2]
ll_nofish = fit_logistic(X[:, :2], frog)[2]
print(f"deviance: null {-2 * ll_null:.1f} on {n - 1} df, model {-2 * ll:.1f} on {n - 3} df")
lr = 2 * (ll - ll_nofish)
print(f"likelihood-ratio test for fish: {lr:.1f} on 1 df, p = {stats.chi2.sf(lr, 1):.1e}")
lpm = np.linalg.lstsq(X, frog, rcond=None)[0] # a straight line fitted to the 0/1 data, for comparison
print(f"linear probability model: fitted 'probabilities' from {np.min(X @ lpm):.2f} to {np.max(X @ lpm):.2f}")Newton's method starts from (every pond 50:50, log-likelihood ) and has converged to four decimals by step 4. scikit-learn agrees to every digit printed.
Reading the coefficients:
- Area. The coefficient 1.673 means that a pond ten times larger has log-odds of frogs 1.673 higher, so its odds are multiplied by (95% CI 2.67 to 10.65), with fish status held fixed.
- Fish. The odds ratio 0.18 (95% CI 0.09 to 0.39) says that, among ponds of the same size, the odds of frogs where there are predatory fish are about one-fifth of the odds where there are none.
- Probabilities. Odds ratios are constant, but changes in probability are not. For a 300 m² pond without fish the model gives ; adding fish gives , a drop of 32 percentage points. For a pond where frogs are almost certain, or almost absent, the same odds ratio moves the probability far less. To communicate effects, compute predicted probabilities for typical cases.
The deviance generalises the residual sum of squares: , where the saturated model fits every observation perfectly (for 0/1 data its log-likelihood is 0, so ). The null model, with an intercept only, has deviance 257.7; the full model 211.5. For nested models, the drop in deviance is the likelihood-ratio statistic, approximately with as many degrees of freedom as coefficients removed. Dropping fish raises the deviance by 23.1 on 1 degree of freedom, . Unlike the residual sum of squares, a deviance on its own says little about fit for binary data; it is the differences between nested models that carry information.
The last line is the reason for all this. A straight line fitted to the 0/1 responses, the "linear probability model", gives fitted values from −0.25 to 0.91: impossible as probabilities at one end. Economists sometimes use it anyway for its simple coefficients, with robust standard errors, when fitted values stay well inside (0, 1); outside that range it misleads.
Poisson regression for collision counts
Now the junctions. The count at junction is modelled as Poisson with mean , and the log link makes effects multiplicative:
where is the traffic through the junction, in millions of vehicles. The term has no coefficient: it is an offset, fixed at 1, so that the model describes the rate , collisions per million vehicles, as . Without the offset, busier junctions would look more dangerous simply because more vehicles pass through them. Offsets appear wherever counts come with different exposures: cases per population at risk, claims per policy-year, species per hectare surveyed, defects per square metre of fabric.
The fitting algorithm is the same Newton iteration with , since the Poisson variance equals its mean.
import numpy as np
# Synthetic road data: collisions in one year at 80 junctions, with traffic (millions of vehicles).
rng = np.random.default_rng(5)
n = 80
traffic = rng.uniform(0.5, 8, n)
roundabout = (rng.random(n) < 0.4).astype(float)
hidden = rng.gamma(3, 1 / 3, n) # unmeasured differences between junctions (mean 1)
collisions = rng.poisson(traffic * np.exp(np.log(1.5) + np.log(0.6) * roundabout) * hidden)
X = np.column_stack([np.ones(n), roundabout])
offset = np.log(traffic) # model a rate: collisions per million vehicles
def fit_poisson(X, y, offset):
"""Newton's method (= iteratively reweighted least squares) for Poisson regression with a log link."""
beta = np.zeros(X.shape[1])
beta[0] = np.log(y.sum() / np.exp(offset).sum()) # start at the overall rate
for _ in range(50):
mu = np.exp(X @ beta + offset)
delta = np.linalg.solve(X.T @ (X * mu[:, None]), X.T @ (y - mu))
beta = beta + delta
if np.max(np.abs(delta)) < 1e-10:
break
mu = np.exp(X @ beta + offset)
return beta, np.linalg.inv(X.T @ (X * mu[:, None])), mu
beta, cov, mu = fit_poisson(X, collisions, offset)
se = np.sqrt(np.diag(cov))
with np.errstate(divide="ignore", invalid="ignore"):
term = np.where(collisions > 0, collisions * np.log(collisions / mu), 0.0)
deviance = 2 * np.sum(term - (collisions - mu))
pearson = np.sum((collisions - mu) ** 2 / mu)
phi = pearson / (n - 2)
print(f"collisions per junction: mean {collisions.mean():.1f}, variance {collisions.var(ddof=1):.1f}")
print(f"base rate: {np.exp(beta[0]):.2f} collisions per million vehicles at ordinary junctions")
print(f"roundabout rate ratio: {np.exp(beta[1]):.2f}")
print(f"residual deviance {deviance:.1f} on {n - 2} df; Pearson dispersion {phi:.2f}")
lo, hi = beta[1] - 1.96 * se[1], beta[1] + 1.96 * se[1]
print(f"Poisson 95% CI for the rate ratio: {np.exp(lo):.2f} to {np.exp(hi):.2f}")
lo, hi = beta[1] - 1.96 * se[1] * np.sqrt(phi), beta[1] + 1.96 * se[1] * np.sqrt(phi)
print(f"quasi-Poisson 95% CI (SE x sqrt(phi)): {np.exp(lo):.2f} to {np.exp(hi):.2f}")The fitted base rate is 1.40 collisions per million vehicles at ordinary junctions, and the roundabout rate ratio is 0.66: roundabouts have about a third fewer collisions per vehicle. But one number in the output should stop us: the residual deviance is 221.4 on 78 degrees of freedom, and the Pearson dispersion statistic
is 2.91. If the counts really were Poisson, each term would average about 1 and would be close to 1. (The raw variance of 24.5 against a mean of 5.2 is not itself evidence, because junctions differ in traffic; compares each count with its own fitted mean.)
Overdispersion
Counts that vary more than the Poisson distribution allows are overdispersed, and in practice they are the rule, not the exception. Junctions differ in ways the model does not record (sight lines, speed, lighting, pedestrian crossings), so even junctions with the same traffic and design have different true rates. A mixture of Poisson distributions with different rates has variance larger than its mean. In the synthetic data, each junction's rate was multiplied by a hidden factor with mean 1 and variance , which gives variance instead of .
Overdispersion does not bias the coefficients much (the log-linear mean is still right), but it makes the Poisson standard errors too small, and so the intervals too narrow and the -values too small. How much too small? Simulation answers.
import numpy as np
# How often does a 95% interval for the roundabout effect catch the truth (log 0.6)?
rng = np.random.default_rng(99)
n, true_b1 = 80, np.log(0.6)
traffic = rng.uniform(0.5, 8, n)
roundabout = (rng.random(n) < 0.4).astype(float)
X = np.column_stack([np.ones(n), roundabout])
offset = np.log(traffic)
mu_true = traffic * 1.5 * np.exp(true_b1 * roundabout)
def fit_poisson(X, y, offset):
beta = np.array([np.log(y.sum() / np.exp(offset).sum()), 0.0])
for _ in range(50):
mu = np.exp(X @ beta + offset)
delta = np.linalg.solve(X.T @ (X * mu[:, None]), X.T @ (y - mu))
beta = beta + delta
if np.max(np.abs(delta)) < 1e-10:
break
mu = np.exp(X @ beta + offset)
se = np.sqrt(np.diag(np.linalg.inv(X.T @ (X * mu[:, None]))))
phi = np.sum((y - mu) ** 2 / mu) / (len(y) - X.shape[1])
return beta, se, phi
for label, extra in [("pure Poisson counts", None), ("overdispersed counts", 3.0)]:
naive = quasi = 0
phis = []
for _ in range(2000):
m = mu_true if extra is None else mu_true * rng.gamma(extra, 1 / extra, n)
beta, se, phi = fit_poisson(X, rng.poisson(m), offset)
naive += abs(beta[1] - true_b1) <= 1.96 * se[1]
quasi += abs(beta[1] - true_b1) <= 1.96 * se[1] * np.sqrt(phi)
phis.append(phi)
print(f"{label:>21}: mean dispersion {np.mean(phis):.2f}; coverage Poisson {naive / 2000:.1%}, quasi-Poisson {quasi / 2000:.1%}")There are three standard remedies.
- Quasi-Poisson: keep the Poisson estimates, multiply standard errors by , and use tests rather than tests. Simple and robust; for the junctions, the roundabout interval widens from 0.54–0.81 to 0.47–0.94.
- Negative binomial regression: model the extra variation explicitly, with variance , as a Poisson whose rate has a gamma distribution, which is exactly how the synthetic data were made. It has a full likelihood, so AIC and likelihood-ratio tests remain available.
- Random effects: give each junction (or each group of observations) its own random term, a Poisson generalised linear mixed model, which also handles repeated counts at the same junctions over years. That is the territory of multilevel models.
Binomial data with more than one trial per observation (such as 7 of 20 seeds germinating per tray) can be overdispersed in the same way, and the same remedies apply. A related problem is an excess of zeros: many sites with no collisions at all because some are almost never used. Zero-inflated and hurdle models handle this, but check first whether a negative binomial already fits.
The professional view
In R, glm(frog ~ log_area + fish, family = binomial, data = ponds) fits the logistic regression, and glm(collisions ~ roundabout + offset(log(traffic)), family = quasipoisson, data = junctions) the quasi-Poisson model; MASS::glm.nb fits negative binomial regression. In Python, statsmodels provides the same families and links. Shown for reference, not run here:
# Python with statsmodels (not run here)
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
fit = smf.glm("frog ~ log_area + fish", data=ponds, family=sm.families.Binomial()).fit()
print(fit.summary()) # coefficients, standard errors, deviance
counts = smf.glm("collisions ~ roundabout", data=junctions, family=sm.families.Poisson(),
offset=np.log(junctions["traffic"])).fit(scale="X2") # quasi-Poisson scalePoints that matter in practice:
- scikit-learn penalises by default. Its
LogisticRegressionadds an L2 penalty unless told otherwise (C=1.0by default), which shrinks coefficients towards zero. That is a sensible choice for prediction with many features, but its coefficients are then not the maximum-likelihood estimates, and it reports no standard errors. The program above passedpenalty=Noneto make the comparison fair. - Odds ratios are not risk ratios. When the outcome is rare (under about 10%), the odds ratio approximates the ratio of probabilities; when it is common, the odds ratio is further from 1. Medical journals increasingly ask for risk differences and risk ratios, which can be computed from a fitted logistic model as averages of predicted probabilities over the sample.
- Case-control studies. In medicine, a study may sample people with a disease and people without it, fixing the outcome proportions by design. Remarkably, logistic regression still estimates the odds ratios of the exposures correctly (only the intercept is distorted), which is a large part of why it dominates epidemiology.
- Beyond straight lines. Generalised additive models replace linear terms by smooth curves; GLMs with random effects handle grouped data; and the same framework, with a different loss, is the output layer of most neural-network classifiers, where the logistic function appears as the sigmoid. See Lumenary CS's learning from data.
- Calibration and fairness. When logistic models score people, for credit, hiring or clinical risk, check that predicted probabilities match observed frequencies in each group the model affects, and that the predictors used are lawful and justifiable. A well-fitting model can still encode an unjust history.
Common misconceptions
Misconception
"A logistic coefficient of 1.67 means the probability rises by 1.67, or by 167%." The coefficient is a change in log-odds. It multiplies the odds by , and the change in probability depends on where you start: largest near , small near 0 or 1.
Misconception
"An odds ratio of 0.18 means the event is 82% less likely." That would be a risk ratio. Odds ratios and risk ratios agree only when the event is rare. In the pond example, fish took a 300 m² pond's probability of frogs from 0.46 to 0.13, a risk ratio of about 0.29, not 0.18.
Misconception
"Poisson regression is fine as long as the data are counts." The Poisson model also asserts that the variance equals the mean, given the predictors. Real counts are usually overdispersed, and ignoring it makes intervals too narrow. Always compute the dispersion statistic.
Misconception
"A large residual deviance means a poor model, and the deviance is like ." For binary responses the residual deviance cannot measure absolute fit, because the saturated model is degenerate. Compare nested models by differences in deviance, and judge fit with residual plots grouped by fitted probability or with calibration checks.
Exercises
1. Generalised linear models
Warm-upHumanWhy is ordinary linear regression a poor model for a yes-or-no outcome, such as whether a seedling survives its first dry season? Select all that apply.
2. Generalised linear models
Warm-upHumanIn a synthetic survey, the probability that a household in a district of Accra has a solar panel is 0.8. What are the odds of having one (written as a single number, "odds of … to 1")?
3. Generalised linear models
CoreHumanA logistic regression of whether a patient's malaria test is positive includes the indicator "slept under an insecticide-treated net", with coefficient (other predictors held fixed; synthetic data). Which statement is correct?
4. Generalised linear models
CoreHumanA Poisson regression with offset estimates a collision rate of collisions per million vehicles at ordinary junctions and a roundabout rate ratio of . How many collisions per year does the model expect at a roundabout carrying 6 million vehicles a year?
5. Generalised linear models
CoreHumanA Poisson regression of the number of insurance claims for 64 groups of drivers, with 4 coefficients, gives a Pearson statistic . A quasi-Poisson analysis keeps the coefficients. What should it do to their standard errors?
6. Generalised linear models
StretchHumanIn a synthetic study, 40% of pupils who do not attend a revision programme pass an exam. A logistic regression gives an odds ratio of 2 for attending. What is the model's pass probability for attenders (with the same other characteristics)?
7. Generalised linear models
CoreComputationalComplete fit_logistic(X, y), which fits a logistic regression by Newton's method and returns the coefficient vector. X is the design matrix (with its column of ones) and y holds 0s and 1s. At each step compute , the score and the information with , and update ; stop when the step is tiny. The tests check that the score is zero at your answer and that it is a maximum.
8. Generalised linear models
CoreComputationalWrite two functions for checking a Poisson regression:
poisson_deviance(y, mu)returns , where a term with is just (since );pearson_dispersion(y, mu, p)returns for a model with coefficients.
Further reading and resources
- Generalized linear models · J. A. Nelder and R. W. M. Wedderburn · 1972Journal of the Royal Statistical Society, Series A, volume 135. The paper that unified regression for many kinds of response.
- R documentation: glm (fitting generalised linear models)
- statsmodels: generalized linear modelsFamilies, links, deviance and robust standard errors in Python.
- scikit-learn: logistic regressionNote that the default fit is penalised (regularised); see the professional view.
- Regression and Other Stories · Andrew Gelman, Jennifer Hill and Aki Vehtari · 2020Data and code for the book's chapters on logistic and count regression, which are strong on interpretation.
- NIST/SEMATECH e-Handbook of Statistical Methods · National Institute of Standards and Technology