Skip to content
LumenaryAcademy
Modelling 75 min

Generalised linear models

Regression for yes-or-no outcomes and counts — logistic regression fitted by Newton's method, odds ratios, Poisson regression with exposures, link functions, deviance, and what to do when counts vary more than the model allows.

Code tested · 5AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

When the response is yes, no, or how many

Wanjiru is an ecologist surveying 200 ponds in the highlands of Kenya for a species of frog. For each pond she records one thing about the frogs, present or absent, and two things about the pond: its area, and whether predatory fish live in it. A few hundred kilometres away, a road-safety engineer in Nairobi is studying 80 junctions: for each, how many collisions happened last year, how much traffic passed through, and whether the junction is a roundabout. (Both datasets in this lesson are synthetic, generated by programs that mimic real surveys.)

Both want what regression offers: how does the response change with the predictors, and how sure are we? But neither response is a continuous measurement with constant scatter around a line.

  • A yes-or-no outcome has a probability, which must stay between 0 and 1. A straight line will eventually leave that range; fitted to Wanjiru's data it predicts a "probability" of −0.25 for small ponds with fish, as we shall see.
  • A count cannot be negative, and its variance grows with its mean: a junction that averages 2 collisions a year varies by a collision or two, one that averages 20 by four or five. Effects are naturally multiplicative too: a roundabout might cut collisions by a third, whatever the traffic.

Generalised linear models (GLMs) keep everything good about linear regression (a linear combination of predictors, coefficients with standard errors, tests of nested models) and change two things: the distribution of the response, and the scale on which the predictors act. John Nelder and Robert Wedderburn, working at Rothamsted Experimental Station in England, unified these models in 1972, showing that one fitting algorithm serves them all. Logistic regression is now the standard tool for binary outcomes in medicine, ecology and the social sciences, and Poisson regression for counts in epidemiology, insurance and traffic safety.

DefinitionsH

Three ingredients

Definition— generalised linear model

A GLM has three parts:

  1. a random component: given the predictors, YiY_i has a distribution from the exponential family (normal, binomial, Poisson, gamma, …) with mean μi\mu_i;
  2. a linear predictor: ηi=β0+β1xi1+⋯+βp−1xi,p−1\eta_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_{p-1} x_{i,p-1}, or η=Xβ\boldsymbol\eta = X\boldsymbol\beta;
  3. a link function gg, smooth and monotone, connecting them: g(μi)=ηig(\mu_i) = \eta_i, so μi=g−1(ηi)\mu_i = g^{-1}(\eta_i).
predictorsx₁, x₂, …linear predictorη = Xβmeanμ = g⁻¹(η)distribution of Ybinomial, Poisson, …any real numberin the allowed rangevariance tied to μlogistic: μ = 1/(1 + e^(−η)) · Poisson: μ = e^η · normal: μ = η
The three ingredients of a generalised linear model. The predictors combine linearly into η, which can be any real number; the inverse link maps η to a mean in the range the response allows; and the distribution of the response, whose variance depends on its mean, describes the scatter around that mean.

The link stretches the restricted range of the mean onto the whole real line, where a linear predictor can roam freely. The common choices:

ResponseDistributionVarianceUsual link g(μ)g(\mu)Model name
MeasurementNormalσ2\sigma^2identity, μ\mulinear regression
Yes/no, or kk of mmBinomialμ(1−μ)\mu(1-\mu) per triallogit, log⁡μ1−μ\log\frac{\mu}{1-\mu}logistic regression
CountPoissonμ\mulog, log⁡μ\log \muPoisson regression
Positive, skewed (costs, durations)Gammaϕμ2\phi\mu^2loggamma regression

The distributions themselves are described in distributions and, with their mathematics, in Lumenary Math's probability theory. Each distribution has a canonical link that makes the mathematics especially tidy (logit for the binomial, log for the Poisson, identity for the normal), but other links are legitimate. For binary data the probit link, the inverse of the standard normal distribution function, gives almost the same fits as the logit and dates back to bioassay in the 1930s; the complementary log-log link suits probabilities that approach 1 asymmetrically. Linear regression is the GLM with a normal distribution and the identity link, so everything here generalises the previous two lessons.

For a probability pp, the odds are p/(1−p)p/(1-p): a probability of 0.8 is odds of 4 (to 1). The logit is the log of the odds. In logistic regression

log⁡pi1−pi=β0+β1xi1+⋯ ,pi=11+e−ηi,\log\frac{p_i}{1 - p_i} = \beta_0 + \beta_1 x_{i1} + \cdots, \qquad p_i = \frac{1}{1 + e^{-\eta_i}},

so increasing xjx_j by one unit adds βj\beta_j to the log-odds, which multiplies the odds by eβje^{\beta_j}, the odds ratio. In Poisson regression with the log link, a unit increase in xjx_j multiplies the expected count by eβje^{\beta_j}, the rate ratio.

ExplanationH

Fitting by maximum likelihood and Newton's method

Least squares no longer fits the bill: the variance changes from observation to observation, and the mean is a curve. GLMs are fitted by maximum likelihood, choosing the coefficients that make the observed data most probable (the general theory is in likelihood and model selection). For logistic regression with yi∈{0,1}y_i \in \{0, 1\}, the probability of the data is ∏piyi(1−pi)1−yi\prod p_i^{y_i}(1-p_i)^{1-y_i}, and its logarithm, after a line of algebra, is

ℓ(β)=∑i=1n[ yiηi−log⁡(1+eηi) ],ηi=xi⊤β.\ell(\boldsymbol\beta) = \sum_{i=1}^n \big[\, y_i \eta_i - \log(1 + e^{\eta_i}) \,\big], \qquad \eta_i = \mathbf x_i^\top \boldsymbol\beta.

Theorem— Score, information and Newton's method for logistic regression

The gradient (score) and Hessian of the logistic log-likelihood are

∇ℓ(β)=X⊤(y−p),∇2ℓ(β)=−X⊤WX,W=diag⁡(pi(1−pi)).\nabla \ell(\boldsymbol\beta) = X^\top(\mathbf y - \mathbf p), \qquad \nabla^2 \ell(\boldsymbol\beta) = -X^\top W X, \quad W = \operatorname{diag}\big(p_i(1 - p_i)\big).

The Hessian is negative definite when XX has full column rank, so ℓ\ell is strictly concave and has at most one maximum. Newton's method updates

β←β+(X⊤WX)−1X⊤(y−p),\boldsymbol\beta \leftarrow \boldsymbol\beta + (X^\top W X)^{-1} X^\top (\mathbf y - \mathbf p),

and at the maximum, (X⊤WX)−1(X^\top W X)^{-1} estimates the covariance matrix of β^\hat{\boldsymbol\beta}.

Proof

Since ddηlog⁡(1+eη)=eη1+eη=p\frac{d}{d\eta}\log(1 + e^{\eta}) = \frac{e^\eta}{1 + e^\eta} = p, the derivative of the ii-th term with respect to β\boldsymbol\beta is (yi−pi)xi(y_i - p_i)\mathbf x_i; summing gives X⊤(y−p)X^\top(\mathbf y - \mathbf p). Differentiating again, dpdη=p(1−p)\frac{dp}{d\eta} = p(1-p), so the Hessian is −∑pi(1−pi)xixi⊤=−X⊤WX-\sum p_i(1-p_i)\mathbf x_i \mathbf x_i^\top = -X^\top W X. For any v≠0\mathbf v \neq \mathbf 0, v⊤X⊤WXv=∑pi(1−pi)(xi⊤v)2>0\mathbf v^\top X^\top W X \mathbf v = \sum p_i(1-p_i)(\mathbf x_i^\top \mathbf v)^2 > 0, because every weight is positive and Xv≠0X\mathbf v \neq \mathbf 0 by full rank. Newton's method replaces ℓ\ell by its second-order Taylor expansion at the current β\boldsymbol\beta and jumps to that quadratic's maximum, which is the update shown. The covariance statement is the general large-sample result that the inverse of the Fisher information, here X⊤WXX^\top W X, is the asymptotic covariance of a maximum-likelihood estimator.

∎

The update can be rewritten as a weighted least-squares fit of a "working response" zi=ηi+(yi−pi)/wiz_i = \eta_i + (y_i - p_i)/w_i with weights wiw_i, so the algorithm is also called iteratively reweighted least squares (IRLS). It is how R's glm and statsmodels fit every GLM: for any canonical link, the score is X⊤(y−μ)X^\top(\mathbf y - \boldsymbol\mu) and the information is X⊤WXX^\top W X with WW the variance function evaluated at μ\boldsymbol\mu, so one short loop fits them all. Newton's method converges quadratically near the maximum, roughly doubling the number of correct digits each step; the numerical analysis is in Lumenary Math's numerical methods.

ComputationC

Logistic regression for the frog survey

The program generates Wanjiru's synthetic survey, fits the logistic regression by Newton's method from scratch, and checks the coefficients with scikit-learn, an independent implementation that uses a different optimiser. Pond area enters as log⁡10\log_{10} of the area in square metres, centred at 2.5 (about 300 m²), so one unit means a pond ten times larger.

python
import numpy as np
from scipy import stats
from sklearn.linear_model import LogisticRegression

# Synthetic survey of 200 ponds: is a frog species present (1) or not (0)?
rng = np.random.default_rng(12)
n = 200
log_area = rng.normal(2.5, 0.6, n)                 # log10 of pond area in m^2 (roughly 30 to 3000 m^2)
fish = (rng.random(n) < 0.45).astype(float)        # predatory fish present?
eta = -0.5 + 2.0 * (log_area - 2.5) - 1.5 * fish   # the true log-odds in this synthetic world
frog = (rng.random(n) < 1 / (1 + np.exp(-eta))).astype(float)
X = np.column_stack([np.ones(n), log_area - 2.5, fish])

def fit_logistic(X, y, verbose=False):
    """Newton's method on the logistic log-likelihood. Returns coefficients, covariance, log-likelihood."""
    beta = np.zeros(X.shape[1])
    for step in range(50):
        p = 1 / (1 + np.exp(-X @ beta))
        loglik = np.sum(y * np.log(p) + (1 - y) * np.log(1 - p))
        if verbose and step < 5:
            print(f"step {step}: beta = {np.round(beta, 4)}, log-likelihood {loglik:.4f}")
        score = X.T @ (y - p)                          # gradient of the log-likelihood
        info = X.T @ (X * (p * (1 - p))[:, None])      # minus the Hessian: X^T W X
        delta = np.linalg.solve(info, score)
        beta = beta + delta
        if np.max(np.abs(delta)) < 1e-10:
            break
    p = 1 / (1 + np.exp(-X @ beta))
    loglik = np.sum(y * np.log(p) + (1 - y) * np.log(1 - p))
    return beta, np.linalg.inv(X.T @ (X * (p * (1 - p))[:, None])), loglik

beta, cov, ll = fit_logistic(X, frog, verbose=True)
se = np.sqrt(np.diag(cov))
sk = LogisticRegression(penalty=None, tol=1e-10, max_iter=1000).fit(X[:, 1:], frog)
print("Newton:      ", np.round(beta, 4))
print("scikit-learn:", np.round(np.r_[sk.intercept_, sk.coef_[0]], 4))
for name, b, s in zip(["log10 area", "fish"], beta[1:], se[1:]):
    print(f"{name:>10}: coefficient {b:+.3f} (SE {s:.3f}), odds ratio {np.exp(b):.2f}, "
          f"95% CI {np.exp(b - 1.96 * s):.2f} to {np.exp(b + 1.96 * s):.2f}")

# Deviances and a likelihood-ratio test for fish
ll_null = fit_logistic(X[:, :1], frog)[2]
ll_nofish = fit_logistic(X[:, :2], frog)[2]
print(f"deviance: null {-2 * ll_null:.1f} on {n - 1} df, model {-2 * ll:.1f} on {n - 3} df")
lr = 2 * (ll - ll_nofish)
print(f"likelihood-ratio test for fish: {lr:.1f} on 1 df, p = {stats.chi2.sf(lr, 1):.1e}")

lpm = np.linalg.lstsq(X, frog, rcond=None)[0]       # a straight line fitted to the 0/1 data, for comparison
print(f"linear probability model: fitted 'probabilities' from {np.min(X @ lpm):.2f} to {np.max(X @ lpm):.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Newton's method starts from β=0\boldsymbol\beta = \mathbf 0 (every pond 50:50, log-likelihood 200log⁡12=−138.63200\log\tfrac12 = -138.63) and has converged to four decimals by step 4. scikit-learn agrees to every digit printed.

Reading the coefficients:

  • Area. The coefficient 1.673 means that a pond ten times larger has log-odds of frogs 1.673 higher, so its odds are multiplied by e1.673=5.33e^{1.673} = 5.33 (95% CI 2.67 to 10.65), with fish status held fixed.
  • Fish. The odds ratio 0.18 (95% CI 0.09 to 0.39) says that, among ponds of the same size, the odds of frogs where there are predatory fish are about one-fifth of the odds where there are none.
  • Probabilities. Odds ratios are constant, but changes in probability are not. For a 300 m² pond without fish the model gives p=1/(1+e0.1748)=0.46p = 1/(1 + e^{0.1748}) = 0.46; adding fish gives 1/(1+e0.1748+1.704)=0.131/(1 + e^{0.1748 + 1.704}) = 0.13, a drop of 32 percentage points. For a pond where frogs are almost certain, or almost absent, the same odds ratio moves the probability far less. To communicate effects, compute predicted probabilities for typical cases.

The deviance generalises the residual sum of squares: D=2(ℓsaturated−ℓmodel)D = 2(\ell_{\text{saturated}} - \ell_{\text{model}}), where the saturated model fits every observation perfectly (for 0/1 data its log-likelihood is 0, so D=−2ℓD = -2\ell). The null model, with an intercept only, has deviance 257.7; the full model 211.5. For nested models, the drop in deviance is the likelihood-ratio statistic, approximately χ2\chi^2 with as many degrees of freedom as coefficients removed. Dropping fish raises the deviance by 23.1 on 1 degree of freedom, p=1.5×10−6p = 1.5 \times 10^{-6}. Unlike the residual sum of squares, a deviance on its own says little about fit for binary data; it is the differences between nested models that carry information.

The last line is the reason for all this. A straight line fitted to the 0/1 responses, the "linear probability model", gives fitted values from −0.25 to 0.91: impossible as probabilities at one end. Economists sometimes use it anyway for its simple coefficients, with robust standard errors, when fitted values stay well inside (0, 1); outside that range it misleads.

10100100010 00000.250.50.751pond area (m², log scale)probability frogs presentno fish (open circles)fish present (filled dots)
The fitted logistic curves for the synthetic pond survey: the probability that frogs are present against pond area (on a log scale), for ponds without fish (solid) and with fish (dashed). The dots along the top are ponds where frogs were found and those along the bottom ponds where they were not. On the log-odds scale the two curves are parallel lines 1.70 apart; on the probability scale they are S-shaped and furthest apart in the middle.
ComputationC

Poisson regression for collision counts

Now the junctions. The count at junction ii is modelled as Poisson with mean μi\mu_i, and the log link makes effects multiplicative:

log⁡μi=log⁡ti+β0+β1 roundabouti,\log \mu_i = \log t_i + \beta_0 + \beta_1\,\text{roundabout}_i,

where tit_i is the traffic through the junction, in millions of vehicles. The term log⁡ti\log t_i has no coefficient: it is an offset, fixed at 1, so that the model describes the rate μi/ti\mu_i/t_i, collisions per million vehicles, as eβ0+β1roundaboutie^{\beta_0 + \beta_1\text{roundabout}_i}. Without the offset, busier junctions would look more dangerous simply because more vehicles pass through them. Offsets appear wherever counts come with different exposures: cases per population at risk, claims per policy-year, species per hectare surveyed, defects per square metre of fabric.

The fitting algorithm is the same Newton iteration with W=diag⁡(μi)W = \operatorname{diag}(\mu_i), since the Poisson variance equals its mean.

python
import numpy as np

# Synthetic road data: collisions in one year at 80 junctions, with traffic (millions of vehicles).
rng = np.random.default_rng(5)
n = 80
traffic = rng.uniform(0.5, 8, n)
roundabout = (rng.random(n) < 0.4).astype(float)
hidden = rng.gamma(3, 1 / 3, n)                     # unmeasured differences between junctions (mean 1)
collisions = rng.poisson(traffic * np.exp(np.log(1.5) + np.log(0.6) * roundabout) * hidden)
X = np.column_stack([np.ones(n), roundabout])
offset = np.log(traffic)                            # model a rate: collisions per million vehicles

def fit_poisson(X, y, offset):
    """Newton's method (= iteratively reweighted least squares) for Poisson regression with a log link."""
    beta = np.zeros(X.shape[1])
    beta[0] = np.log(y.sum() / np.exp(offset).sum())   # start at the overall rate
    for _ in range(50):
        mu = np.exp(X @ beta + offset)
        delta = np.linalg.solve(X.T @ (X * mu[:, None]), X.T @ (y - mu))
        beta = beta + delta
        if np.max(np.abs(delta)) < 1e-10:
            break
    mu = np.exp(X @ beta + offset)
    return beta, np.linalg.inv(X.T @ (X * mu[:, None])), mu

beta, cov, mu = fit_poisson(X, collisions, offset)
se = np.sqrt(np.diag(cov))
with np.errstate(divide="ignore", invalid="ignore"):
    term = np.where(collisions > 0, collisions * np.log(collisions / mu), 0.0)
deviance = 2 * np.sum(term - (collisions - mu))
pearson = np.sum((collisions - mu) ** 2 / mu)
phi = pearson / (n - 2)
print(f"collisions per junction: mean {collisions.mean():.1f}, variance {collisions.var(ddof=1):.1f}")
print(f"base rate: {np.exp(beta[0]):.2f} collisions per million vehicles at ordinary junctions")
print(f"roundabout rate ratio: {np.exp(beta[1]):.2f}")
print(f"residual deviance {deviance:.1f} on {n - 2} df; Pearson dispersion {phi:.2f}")
lo, hi = beta[1] - 1.96 * se[1], beta[1] + 1.96 * se[1]
print(f"Poisson 95% CI for the rate ratio:       {np.exp(lo):.2f} to {np.exp(hi):.2f}")
lo, hi = beta[1] - 1.96 * se[1] * np.sqrt(phi), beta[1] + 1.96 * se[1] * np.sqrt(phi)
print(f"quasi-Poisson 95% CI (SE x sqrt(phi)):  {np.exp(lo):.2f} to {np.exp(hi):.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The fitted base rate is 1.40 collisions per million vehicles at ordinary junctions, and the roundabout rate ratio is 0.66: roundabouts have about a third fewer collisions per vehicle. But one number in the output should stop us: the residual deviance is 221.4 on 78 degrees of freedom, and the Pearson dispersion statistic

ϕ^=1n−p∑i(yi−μ^i)2μ^i\hat\phi = \frac{1}{n - p}\sum_i \frac{(y_i - \hat\mu_i)^2}{\hat\mu_i}

is 2.91. If the counts really were Poisson, each term would average about 1 and ϕ^\hat\phi would be close to 1. (The raw variance of 24.5 against a mean of 5.2 is not itself evidence, because junctions differ in traffic; ϕ^\hat\phi compares each count with its own fitted mean.)

ComputationC

Overdispersion

Counts that vary more than the Poisson distribution allows are overdispersed, and in practice they are the rule, not the exception. Junctions differ in ways the model does not record (sight lines, speed, lighting, pedestrian crossings), so even junctions with the same traffic and design have different true rates. A mixture of Poisson distributions with different rates has variance larger than its mean. In the synthetic data, each junction's rate was multiplied by a hidden factor with mean 1 and variance 1/31/3, which gives variance μ+μ2/3\mu + \mu^2/3 instead of μ\mu.

Overdispersion does not bias the coefficients much (the log-linear mean is still right), but it makes the Poisson standard errors too small, and so the intervals too narrow and the pp-values too small. How much too small? Simulation answers.

python
import numpy as np

# How often does a 95% interval for the roundabout effect catch the truth (log 0.6)?
rng = np.random.default_rng(99)
n, true_b1 = 80, np.log(0.6)
traffic = rng.uniform(0.5, 8, n)
roundabout = (rng.random(n) < 0.4).astype(float)
X = np.column_stack([np.ones(n), roundabout])
offset = np.log(traffic)
mu_true = traffic * 1.5 * np.exp(true_b1 * roundabout)

def fit_poisson(X, y, offset):
    beta = np.array([np.log(y.sum() / np.exp(offset).sum()), 0.0])
    for _ in range(50):
        mu = np.exp(X @ beta + offset)
        delta = np.linalg.solve(X.T @ (X * mu[:, None]), X.T @ (y - mu))
        beta = beta + delta
        if np.max(np.abs(delta)) < 1e-10:
            break
    mu = np.exp(X @ beta + offset)
    se = np.sqrt(np.diag(np.linalg.inv(X.T @ (X * mu[:, None]))))
    phi = np.sum((y - mu) ** 2 / mu) / (len(y) - X.shape[1])
    return beta, se, phi

for label, extra in [("pure Poisson counts", None), ("overdispersed counts", 3.0)]:
    naive = quasi = 0
    phis = []
    for _ in range(2000):
        m = mu_true if extra is None else mu_true * rng.gamma(extra, 1 / extra, n)
        beta, se, phi = fit_poisson(X, rng.poisson(m), offset)
        naive += abs(beta[1] - true_b1) <= 1.96 * se[1]
        quasi += abs(beta[1] - true_b1) <= 1.96 * se[1] * np.sqrt(phi)
        phis.append(phi)
    print(f"{label:>21}: mean dispersion {np.mean(phis):.2f}; coverage Poisson {naive / 2000:.1%}, quasi-Poisson {quasi / 2000:.1%}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

There are three standard remedies.

  • Quasi-Poisson: keep the Poisson estimates, multiply standard errors by ϕ^\sqrt{\hat\phi}, and use FF tests rather than χ2\chi^2 tests. Simple and robust; for the junctions, the roundabout interval widens from 0.54–0.81 to 0.47–0.94.
  • Negative binomial regression: model the extra variation explicitly, with variance μ+μ2/k\mu + \mu^2/k, as a Poisson whose rate has a gamma distribution, which is exactly how the synthetic data were made. It has a full likelihood, so AIC and likelihood-ratio tests remain available.
  • Random effects: give each junction (or each group of observations) its own random term, a Poisson generalised linear mixed model, which also handles repeated counts at the same junctions over years. That is the territory of multilevel models.

Binomial data with more than one trial per observation (such as 7 of 20 seeds germinating per tray) can be overdispersed in the same way, and the same remedies apply. A related problem is an excess of zeros: many sites with no collisions at all because some are almost never used. Zero-inflated and hurdle models handle this, but check first whether a negative binomial already fits.

ExplanationH

The professional view

In R, glm(frog ~ log_area + fish, family = binomial, data = ponds) fits the logistic regression, and glm(collisions ~ roundabout + offset(log(traffic)), family = quasipoisson, data = junctions) the quasi-Poisson model; MASS::glm.nb fits negative binomial regression. In Python, statsmodels provides the same families and links. Shown for reference, not run here:

python
# Python with statsmodels (not run here)
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
fit = smf.glm("frog ~ log_area + fish", data=ponds, family=sm.families.Binomial()).fit()
print(fit.summary())                  # coefficients, standard errors, deviance
counts = smf.glm("collisions ~ roundabout", data=junctions, family=sm.families.Poisson(),
                 offset=np.log(junctions["traffic"])).fit(scale="X2")   # quasi-Poisson scale

Points that matter in practice:

  • scikit-learn penalises by default. Its LogisticRegression adds an L2 penalty unless told otherwise (C=1.0 by default), which shrinks coefficients towards zero. That is a sensible choice for prediction with many features, but its coefficients are then not the maximum-likelihood estimates, and it reports no standard errors. The program above passed penalty=None to make the comparison fair.
  • Odds ratios are not risk ratios. When the outcome is rare (under about 10%), the odds ratio approximates the ratio of probabilities; when it is common, the odds ratio is further from 1. Medical journals increasingly ask for risk differences and risk ratios, which can be computed from a fitted logistic model as averages of predicted probabilities over the sample.
  • Case-control studies. In medicine, a study may sample people with a disease and people without it, fixing the outcome proportions by design. Remarkably, logistic regression still estimates the odds ratios of the exposures correctly (only the intercept is distorted), which is a large part of why it dominates epidemiology.
  • Beyond straight lines. Generalised additive models replace linear terms by smooth curves; GLMs with random effects handle grouped data; and the same framework, with a different loss, is the output layer of most neural-network classifiers, where the logistic function appears as the sigmoid. See Lumenary CS's learning from data.
  • Calibration and fairness. When logistic models score people, for credit, hiring or clinical risk, check that predicted probabilities match observed frequencies in each group the model affects, and that the predictors used are lawful and justifiable. A well-fitting model can still encode an unjust history.
Common misconceptionsH

Common misconceptions

Misconception

"A logistic coefficient of 1.67 means the probability rises by 1.67, or by 167%." The coefficient is a change in log-odds. It multiplies the odds by e1.67≈5.3e^{1.67} \approx 5.3, and the change in probability depends on where you start: largest near p=0.5p = 0.5, small near 0 or 1.

Misconception

"An odds ratio of 0.18 means the event is 82% less likely." That would be a risk ratio. Odds ratios and risk ratios agree only when the event is rare. In the pond example, fish took a 300 m² pond's probability of frogs from 0.46 to 0.13, a risk ratio of about 0.29, not 0.18.

Misconception

"Poisson regression is fine as long as the data are counts." The Poisson model also asserts that the variance equals the mean, given the predictors. Real counts are usually overdispersed, and ignoring it makes intervals too narrow. Always compute the dispersion statistic.

Misconception

"A large residual deviance means a poor model, and the deviance is like 1−R21 - R^2." For binary responses the residual deviance cannot measure absolute fit, because the saturated model is degenerate. Compare nested models by differences in deviance, and judge fit with residual plots grouped by fitted probability or with calibration checks.

Exercises

Exercises

1. Generalised linear models

Warm-upHuman

Why is ordinary linear regression a poor model for a yes-or-no outcome, such as whether a seedling survives its first dry season? Select all that apply.

Choose all that apply

2. Generalised linear models

Warm-upHuman

In a synthetic survey, the probability that a household in a district of Accra has a solar panel is 0.8. What are the odds of having one (written as a single number, "odds of … to 1")?

3. Generalised linear models

CoreHuman

A logistic regression of whether a patient's malaria test is positive includes the indicator "slept under an insecticide-treated net", with coefficient −0.70-0.70 (other predictors held fixed; synthetic data). Which statement is correct?

Choose one

4. Generalised linear models

CoreHuman

A Poisson regression with offset log⁡(traffic)\log(\text{traffic}) estimates a collision rate of eβ0=2e^{\beta_0} = 2 collisions per million vehicles at ordinary junctions and a roundabout rate ratio of eβ1=0.5e^{\beta_1} = 0.5. How many collisions per year does the model expect at a roundabout carrying 6 million vehicles a year?

5. Generalised linear models

CoreHuman

A Poisson regression of the number of insurance claims for 64 groups of drivers, with 4 coefficients, gives a Pearson statistic ∑(yi−μ^i)2/μ^i=180\sum (y_i - \hat\mu_i)^2/\hat\mu_i = 180. A quasi-Poisson analysis keeps the coefficients. What should it do to their standard errors?

Choose one

6. Generalised linear models

StretchHuman

In a synthetic study, 40% of pupils who do not attend a revision programme pass an exam. A logistic regression gives an odds ratio of 2 for attending. What is the model's pass probability for attenders (with the same other characteristics)?

Choose one

7. Generalised linear models

CoreComputational

Complete fit_logistic(X, y), which fits a logistic regression by Newton's method and returns the coefficient vector. X is the design matrix (with its column of ones) and y holds 0s and 1s. At each step compute p=1/(1+e−Xβ)\mathbf p = 1/(1 + e^{-X\boldsymbol\beta}), the score X⊤(y−p)X^\top(\mathbf y - \mathbf p) and the information X⊤WXX^\top W X with W=diag⁡(pi(1−pi))W = \operatorname{diag}(p_i(1 - p_i)), and update β←β+(X⊤WX)−1X⊤(y−p)\boldsymbol\beta \leftarrow \boldsymbol\beta + (X^\top W X)^{-1}X^\top(\mathbf y - \mathbf p); stop when the step is tiny. The tests check that the score is zero at your answer and that it is a maximum.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Generalised linear models

CoreComputational

Write two functions for checking a Poisson regression:

  • poisson_deviance(y, mu) returns D=2∑i[yilog⁡(yi/μi)−(yi−μi)]D = 2\sum_i \big[y_i \log(y_i/\mu_i) - (y_i - \mu_i)\big], where a term with yi=0y_i = 0 is just 2μi2\mu_i (since 0log⁡0=00 \log 0 = 0);
  • pearson_dispersion(y, mu, p) returns ϕ^=1n−p∑i(yi−μi)2/μi\hat\phi = \frac{1}{n - p}\sum_i (y_i - \mu_i)^2/\mu_i for a model with pp coefficients.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources