Skip to content
LumenaryAcademy
Advanced 80 min

Causal inference

What would have happened otherwise? Potential outcomes, randomisation, causal diagrams, propensity scores, and the natural experiments that won the 2021 economics Nobel prize.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Did the programme work?

A city education office runs free after-school maths tutoring. At the end of the year an analyst compares the pupils who attended with those who did not, and finds that the tutored pupils scored about five marks lower. Should the city close the programme?

Almost certainly not: the pupils who signed up were the ones who were struggling, and would have scored lower with or without tutoring. The comparison mixes what tutoring did with who chose it, and the raw numbers cannot separate the two.

The questions people care about most are causal. Does this drug reduce strokes? Does a higher minimum wage cost jobs? Do smaller classes help children learn? Each asks about a world we did not observe: what would have happened to the same people under a different choice. Regression (Multiple regression and diagnostics) describes how outcomes vary with variables in the world we did observe; causal inference asks when, and under what assumptions, such descriptions answer a "what if" question.

This lesson builds the subject in three layers: a precise language for causal effects (potential outcomes); the design that solves the problem cleanly (randomisation); and the tools for when we cannot randomise: causal diagrams to decide what to adjust for, propensity scores to do the adjusting, and three natural experiments that exploit accidents of nature and policy, whose champions shared the 2021 Nobel prize in economics.

DefinitionsH

Potential outcomes

Give each pupil ii two numbers: Yi(1)Y_i(1), the mark they would get with tutoring, and Yi(0)Y_i(0), the mark they would get without. These are potential outcomes. Both exist, in the sense that each describes a definite possible world, but at most one is ever seen.

Definition— potential outcomes and causal effects

For a binary treatment Ti∈{0,1}T_i \in \{0, 1\}, unit ii has potential outcomes Yi(0)Y_i(0) and Yi(1)Y_i(1), and the observed outcome is Yi=Yi(Ti)Y_i = Y_i(T_i). The individual causal effect is Yi(1)−Yi(0)Y_i(1) - Y_i(0). The average treatment effect is ATE=E[Y(1)−Y(0)]\text{ATE} = E[Y(1) - Y(0)] over the population, and the average treatment effect on the treated is ATT=E[Y(1)−Y(0)∣T=1]\text{ATT} = E[Y(1) - Y(0) \mid T = 1].

The notation rests on the stable unit treatment value assumption (SUTVA): each unit's outcome depends only on its own treatment (no interference), and there is only one version of the treatment. Vaccines, which protect the unvaccinated by slowing transmission, violate the first half; "exercise", which can mean anything from walking to marathons, strains the second.

Here are six pupils, seen with a God's-eye view of both potential outcomes. The three weakest chose tutoring.

PupilY(0)Y(0)Y(1)Y(1)EffectTutored?Observed YY
Amina4857+9yes57
Bo5259+7yes59
Carlos6166+5yes66
Dilnoza7073+3no70
Eero7577+2no75
Farida80800no80

Tutoring helped every pupil or left them unchanged: the ATE is 26/6≈4.3326/6 \approx 4.33 marks and the ATT is 77. Yet the observed comparison is 60.67−75=−14.3360.67 - 75 = -14.33. The algebra of the gap is worth writing out once:

E[Y∣T=1]−E[Y∣T=0]⏟naive: −14.33=E[Y(1)−Y(0)∣T=1]⏟ATT: 7+E[Y(0)∣T=1]−E[Y(0)∣T=0]⏟selection bias: 53.67−75=−21.33.\underbrace{E[Y \mid T{=}1] - E[Y \mid T{=}0]}_{\text{naive: } -14.33} = \underbrace{E[Y(1) - Y(0) \mid T{=}1]}_{\text{ATT: } 7} + \underbrace{E[Y(0) \mid T{=}1] - E[Y(0) \mid T{=}0]}_{\text{selection bias: } 53.67 - 75 = -21.33}.

The selection-bias term compares the two groups in the world without tutoring. It is zero only if the groups would have done equally well untreated, and that is exactly what self-selection destroys. Paul Holland called the fact that we never see both columns for the same unit the fundamental problem of causal inference. Every method in this lesson is a way of filling in the missing column for a group, never for an individual, using an assumption we can state.

ComputationC

Randomisation, the gold standard

Suppose a fair coin decides who is offered tutoring. Then treatment is independent of everything about the pupil, including both potential outcomes:

(Y(0),Y(1))⊥ ⁣ ⁣ ⁣⊥T.\big(Y(0), Y(1)\big) \perp\!\!\!\perp T.

Independence makes the selection-bias term vanish: E[Y(0)∣T=1]=E[Y(0)∣T=0]=E[Y(0)]E[Y(0) \mid T{=}1] = E[Y(0) \mid T{=}0] = E[Y(0)], and likewise for Y(1)Y(1). So the simple difference in group means is an unbiased estimate of the ATE. This is why Ronald Fisher's randomised experiments, which you met in Experimental design and ANOVA, are called the gold standard: they balance not only the variables we measured but every variable, known or unknown, in expectation.

The simulation below creates 2,000 pupils with both potential outcomes, so that we know the truth, and then plays out two worlds: one where pupils choose, and one where a coin decides.

python
import numpy as np

rng = np.random.default_rng(2021)
n = 2000
prior = rng.normal(60, 12, n)                    # last year's maths mark
y0 = 12 + 0.8 * prior + rng.normal(0, 6, n)      # this year's mark WITHOUT tutoring
y1 = y0 + 5 + 0.15 * (60 - prior)                # WITH tutoring: helps weaker pupils more
print(f"true average effect (ATE): {np.mean(y1 - y0):5.2f} marks")

# World A: pupils choose. Weaker pupils are far more likely to sign up.
signs_up = rng.random(n) < 1 / (1 + np.exp((prior - 52) / 5))
y_obs = np.where(signs_up, y1, y0)               # we only ever see one potential outcome
naive = y_obs[signs_up].mean() - y_obs[~signs_up].mean()
att = np.mean((y1 - y0)[signs_up])
print(f"self-selection: {signs_up.mean():.0%} sign up; tutored minus untutored = {naive:6.2f}"
      f"   (effect on those who signed up: {att:.2f})")

# World B: the same pupils, but a fair coin decides who is offered tutoring.
coin = rng.random(n) < 0.5
y_obs = np.where(coin, y1, y0)
se = np.sqrt(y_obs[coin].var(ddof=1) / coin.sum() + y_obs[~coin].var(ddof=1) / (~coin).sum())
print(f"randomised:     tutored minus untutored = {y_obs[coin].mean() - y_obs[~coin].mean():6.2f}"
      f"   (standard error from this one experiment: {se:.2f})")

# Repeat the coin-flipping 10,000 times: the estimate is right on average.
est = np.empty(10_000)
for r in range(est.size):
    c = rng.random(n) < 0.5
    est[r] = y1[c].mean() - y0[~c].mean()
print(f"10,000 randomisations: mean estimate {est.mean():.3f}, standard deviation {est.std():.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The true ATE is 4.94 marks. Under self-selection, the tutored pupils score 5.40 marks lower than the rest, even though tutoring raised every pupil's mark, and raised the marks of those who chose it by 6.59 on average (the ATT is larger than the ATE here because the programme helps weaker pupils more, and they are the ones who chose it). The single randomised experiment gives 5.18 with a standard error of 0.47, comfortably consistent with the truth.

But we cannot randomly assign smoking, schooling or a minimum wage, and sometimes we should not randomise even when we could. The rest of this lesson is about observational data, where people or circumstances chose the treatment, and the assumptions that substitute for the coin.

ExplanationH

Confounding and causal diagrams

Take a second example, with real stakes for the method. A farming cooperative offers maize growers a text-message advisory service with planting and weather advice. Did subscribing raise yields? Farmers with more schooling and bigger plots were more likely to subscribe, and schooling and plot size also raise yields directly. Variables like these, which influence both the treatment and the outcome, are confounders. A causal diagram, or directed acyclic graph (DAG), draws each variable as a node and each direct causal influence as an arrow.

SchoolingPlot areaSubscribes (T)Yield (Y)Fertiliser boughtAnswers surveyconfounderconfounderthe effect we wantmediatorcollider
A causal diagram for the advisory service. Schooling and plot area open back-door paths from subscribing to yield, so they must be adjusted for. Fertiliser bought lies on the causal path (a mediator): adjusting for it would remove part of the effect we want. Answering the follow-up survey is caused by both subscribing and yield (a collider): restricting the analysis to those who answered would create a spurious association.

Association flows along paths in the diagram, whichever way the arrows point. Three small patterns decide whether a path carries association:

  • a fork, T←S→YT \leftarrow S \rightarrow Y: a common cause makes TT and YY associated; conditioning on SS (holding it fixed, stratifying, or including it in a regression) blocks the path;
  • a chain, T→M→YT \rightarrow M \rightarrow Y: the effect flows through the mediator; conditioning on MM blocks it, and removes that part of the causal effect;
  • a collider, T→C←YT \rightarrow C \leftarrow Y: two arrows meet head to head; the path is blocked unless we condition on CC (or on something CC causes), which opens it.

Colliders are the surprise. Suppose farmers were more likely to answer the follow-up survey if they subscribed (they feel engaged) or if their harvest was good (they are happy to talk). Among those who answered, a non-subscriber probably answered because the harvest was good, so non-subscribing respondents have unusually high yields, and the service looks worse than it is. Selecting on a common effect manufactured an association. The same mechanism, Berkson's paradox (after a 1946 study of hospital patients), makes traits that each help someone get selected, for a job, a university place or publication, look like rivals within the selected group.

Definition— backdoor criterion

A set of variables ZZ satisfies the backdoor criterion for the effect of TT on YY if no variable in ZZ is a descendant of TT, and ZZ blocks every path between TT and YY that starts with an arrow into TT (a "back-door" path). Then the effect is identified by the adjustment formula P(Y(t)=y)=∑zP(Y=y∣T=t,Z=z) P(Z=z),P\big(Y(t) = y\big) = \sum_z P(Y = y \mid T = t, Z = z)\,P(Z = z), with an integral for continuous ZZ: compare treated and untreated within levels of ZZ, then average over the population's distribution of ZZ.

In the figure, Z={schooling, plot area}Z = \{\text{schooling, plot area}\} satisfies the criterion. Adding fertiliser bought would violate it (a descendant of TT), and so would restricting to survey respondents. The criterion, due to Judea Pearl, turns the old advice "control for confounders" into a rule that also says what not to control for.

ComputationC

Adjusting for confounders: propensity scores

If the backdoor criterion holds for a set of measured covariates XX, we say treatment is unconfounded given XX: (Y(0),Y(1))⊥ ⁣ ⁣ ⁣⊥T∣X\big(Y(0), Y(1)\big) \perp\!\!\!\perp T \mid X. Within any group of units with the same XX, treatment is as good as random. With one or two discrete covariates we could simply stratify. With many covariates, strata become empty, and we need a summary.

Definition— propensity score

The propensity score is e(x)=P(T=1∣X=x)e(x) = P(T = 1 \mid X = x), the probability of treatment given the covariates. Overlap (or positivity) requires 0<e(x)<10 \lt e(x) \lt 1 for every xx in the population: every kind of unit could have received either treatment.

Theorem— Rosenbaum and Rubin, 1983

If treatment is unconfounded given XX, it is also unconfounded given the single number e(X)e(X): (Y(0),Y(1))⊥ ⁣ ⁣ ⁣⊥T∣e(X)\big(Y(0), Y(1)\big) \perp\!\!\!\perp T \mid e(X).

Proof

It suffices to show that P(T=1∣Y(0),Y(1),e(X))P\big(T = 1 \mid Y(0), Y(1), e(X)\big) does not depend on the potential outcomes. By the tower property, it equals E[P(T=1∣Y(0),Y(1),X)∣Y(0),Y(1),e(X)]E\big[P(T = 1 \mid Y(0), Y(1), X) \mid Y(0), Y(1), e(X)\big]. Unconfoundedness given XX makes the inner probability equal to P(T=1∣X)=e(X)P(T = 1 \mid X) = e(X), and the expectation of e(X)e(X) given e(X)e(X) (and anything else) is e(X)e(X) itself. So the probability of treatment given the potential outcomes and the score is just the score.

∎

So we can balance many covariates by balancing one number. Two standard ways to use it:

  • Inverse probability weighting (IPW) weights each treated unit by 1/e(Xi)1/e(X_i) and each untreated unit by 1/(1−e(Xi))1/(1 - e(X_i)). A treated farmer who was unlikely to subscribe stands in for many similar farmers who did not. The weighted groups each resemble the whole population, and the difference of weighted means estimates the ATE.
  • Matching pairs each treated unit with the untreated unit whose score is closest, and averages the differences within pairs. This estimates the ATT.

The score itself is unknown and must be estimated, usually by logistic regression (Generalised linear models). In the simulation below, the true effect of the service is 0.30 tonnes per hectare for every farmer, and we fit the logistic regression with a few steps of Newton's method so that nothing is hidden in a library.

python
import numpy as np

rng = np.random.default_rng(7)
n = 3000
school = rng.integers(0, 13, n).astype(float)          # years of schooling
area = rng.gamma(2.0, 0.6, n)                          # plot area in hectares
# Better-educated farmers with bigger plots are more likely to subscribe.
logit = -3.0 + 0.25 * school + 0.8 * area
t = rng.random(n) < 1 / (1 + np.exp(-logit))
# Yield (tonnes per hectare): the service adds 0.30 t/ha for everyone.
y = 1.0 + 0.08 * school + 0.25 * area + 0.30 * t + rng.normal(0, 0.4, n)
X = np.column_stack([np.ones(n), school, area])

print(f"{t.mean():.0%} subscribe. Naive difference in mean yield: {y[t].mean() - y[~t].mean():.3f} t/ha (truth 0.300)")

# 1. Propensity scores by logistic regression, fitted with Newton's method.
beta = np.zeros(3)
for _ in range(25):
    p = 1 / (1 + np.exp(-X @ beta))
    grad = X.T @ (t - p)
    hess = X.T @ (X * (p * (1 - p))[:, None])
    beta += np.linalg.solve(hess, grad)
e = 1 / (1 + np.exp(-X @ beta))
print("fitted propensity model:", np.round(beta, 2), " range of scores:", np.round([e.min(), e.max()], 3))

# 2. Inverse probability weighting (normalised weights).
w1, w0 = t / e, (~t) / (1 - e)
ipw = np.sum(w1 * y) / w1.sum() - np.sum(w0 * y) / w0.sum()

# 3. One-to-one nearest-neighbour matching on the logit of the score, with replacement (ATT).
lt, lc = np.log(e[t] / (1 - e[t])), np.log(e[~t] / (1 - e[~t]))
match = np.abs(lt[:, None] - lc[None, :]).argmin(axis=1)
att = np.mean(y[t] - y[~t][match])

# 4. Regression adjustment, for comparison.
coef = np.linalg.lstsq(np.column_stack([X, t]), y, rcond=None)[0]
print(f"IPW: {ipw:.3f}   matching (ATT): {att:.3f}   regression: {coef[3]:.3f}")

# Balance check: standardised mean differences before and after matching.
def smd(a, b):
    return (a.mean() - b.mean()) / np.sqrt((a.var() + b.var()) / 2)
for name, v in [("schooling", school), ("plot area", area)]:
    print(f"  {name:9}  SMD before {smd(v[t], v[~t]):+.2f}   after matching {smd(v[t], v[~t][match]):+.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The naive comparison says 0.625 t/ha, more than double the truth, because subscribers were better placed to begin with. Weighting gives 0.284, matching 0.280 and regression adjustment 0.284, all close to 0.30. The balance table is the step practitioners report first: before matching, subscribers had 0.77 standard deviations more schooling than non-subscribers; after matching, the difference is 0.03. A common rule of thumb treats standardised differences below 0.1 as adequate balance.

One caution about overlap: the estimated scores run from 0.057 to 0.989. A non-subscriber with e=0.989e = 0.989 gets weight 1/0.011≈901/0.011 \approx 90, and a handful of such weights can dominate an IPW estimate. Practitioners inspect the scores in both groups, trim extreme ones, and say which population the trimmed estimate describes.

Finally, the three estimators agree with each other and with the truth only because the program wrote the truth. On real data, agreement among adjustment methods shows that the answer does not hinge on the functional form; it says nothing about unmeasured confounding, which biases all of them alike.

Worked examplesH

Natural experiments

When important confounders are unmeasured, adjustment fails. The alternative is to find a source of variation in the treatment that is as good as random for reasons outside the data: a lottery, a rule, a border, a date. These are natural experiments, and each comes with a design that exploits it.

Instrumental variables

An instrument ZZ is a variable that (1) shifts the treatment (relevance), (2) is as good as randomly assigned (independence), and (3) affects the outcome only through the treatment (exclusion restriction). A randomised encouragement is the cleanest case. Suppose a health service sends a reminder letter, chosen at random, encouraging adults to get a flu vaccine. Health-conscious people get vaccinated anyway and also take fewer sick days, so comparing vaccinated with unvaccinated people is confounded. But the letter is random. The Wald estimator divides the letter's effect on the outcome by its effect on the treatment:

β^IV=YˉZ=1−YˉZ=0TˉZ=1−TˉZ=0.\hat\beta_{\text{IV}} = \frac{\bar Y_{Z=1} - \bar Y_{Z=0}}{\bar T_{Z=1} - \bar T_{Z=0}}.

Guido Imbens and Joshua Angrist showed in 1994 what this ratio estimates when effects differ between people. Divide the population into always-takers (vaccinated regardless), never-takers (never vaccinated) and compliers (vaccinated only if they get the letter), and assume there are no defiers who do the opposite of the encouragement (monotonicity). Then the ratio estimates the local average treatment effect (LATE): the average effect among compliers.

python
import numpy as np

rng = np.random.default_rng(11)

# --- Instrumental variable: a randomly sent reminder letter encourages flu vaccination ---
n = 20_000
kind = rng.choice(["always", "complier", "never"], size=n, p=[0.30, 0.25, 0.45])
z = rng.random(n) < 0.5                                  # letter sent? (random)
d = (kind == "always") | ((kind == "complier") & z)     # vaccinated?
base = np.select([kind == "always", kind == "complier"], [2.0, 3.5], 4.5)   # days off sick if unvaccinated
gain = np.select([kind == "always", kind == "complier"], [1.0, 1.5], 2.0)   # days saved by the vaccine
y = base - gain * d + rng.normal(0, 1.0, n)              # days off work this winter

print(f"vaccinated minus unvaccinated: {y[d].mean() - y[~d].mean():+.2f} days (confounded by health habits)")
itt = y[z].mean() - y[~z].mean()                         # effect of the LETTER
first_stage = d[z].mean() - d[~z].mean()                 # how much the letter moves vaccination
print(f"letter: effect on days {itt:+.3f}, effect on vaccination {first_stage:.3f}")
print(f"Wald (IV) estimate: {itt / first_stage:+.2f} days   "
      f"true effect for compliers {-gain[kind == 'complier'].mean():+.2f}, for everyone {-gain.mean():+.2f}")

# --- Difference-in-differences: free school meals in region A from this year ---
schools = 150
a_before = rng.normal(88.0, 3.0, schools)                # attendance %, region A, last year
b_before = rng.normal(92.0, 3.0, schools)                # region B, last year
a_after = a_before + 1.0 + 2.5 + rng.normal(0, 1.0, schools)   # common trend +1.0, policy +2.5
b_after = b_before + 1.0 + rng.normal(0, 1.0, schools)
print(f"A after minus B after:   {a_after.mean() - b_after.mean():+.2f} points")
print(f"A after minus A before:  {a_after.mean() - a_before.mean():+.2f} points")
did = (a_after.mean() - a_before.mean()) - (b_after.mean() - b_before.mean())
se = np.sqrt(np.var(a_after - a_before, ddof=1) / schools + np.var(b_after - b_before, ddof=1) / schools)
print(f"difference-in-differences: {did:+.2f} points, standard error {se:.2f} (true effect +2.50)")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Comparing vaccinated with unvaccinated people suggests the vaccine saves 2.99 days, double its real effect, because the healthiest people chose it. The letter reduced sick days by only 0.364 on average, but it moved only 24.3% of people into vaccination; dividing gives 1.50 days, exactly the effect for compliers. It is not the average effect for everyone (1.57 days here), because never-takers, who would gain most, are never moved by a letter. An instrument answers a question about the people it moves, which may or may not be the question a policy-maker needs.

The exclusion restriction is the assumption to attack: if the letter also reminded people to wash their hands, the ratio would credit the vaccine with the letter's other effects. And a weak instrument, which barely moves the treatment, divides by a number near zero and amplifies every small bias, which is why studies report the strength of the first stage.

Difference-in-differences

Suppose region A starts free school meals this year and region B does not. Comparing A with B this year mixes the policy with the regions' permanent differences; comparing A this year with A last year mixes the policy with everything else that changed. Difference-in-differences (DiD) subtracts B's change from A's change:

DiD^=(Aˉafter−Aˉbefore)−(Bˉafter−Bˉbefore).\widehat{\text{DiD}} = (\bar A_{\text{after}} - \bar A_{\text{before}}) - (\bar B_{\text{after}} - \bar B_{\text{before}}).

In the simulation, region A has lower attendance throughout, and both regions share an upward trend of one point. The after-only comparison says the policy lowered attendance by 1.34 points; the before–after comparison credits it with the whole rise of 3.53 points, trend included. DiD gives 2.65 with a standard error of 0.11, close to the true 2.50.

86%88%90%92%94%free meals start in region Alast yearthis yearregion B: 93.0region A: 91.6A without meals: 89.0B: 92.1A: 88.1effect ≈ 2.65
Difference-in-differences with the simulated attendance data. The dashed line is region A's counterfactual path, assumed parallel to region B's. The effect is the gap between where region A ended and where the parallel trend says it would have been.

DiD rests on parallel trends: without the policy, A would have changed by the same amount as B. It cannot be tested for the period that matters, but it can be probed: if several years of data before the policy show the two regions moving in step, the assumption is more believable. The classic application is David Card and Alan Krueger's 1994 comparison of fast-food restaurants in New Jersey, which raised its minimum wage, and neighbouring eastern Pennsylvania, which did not. They found no evidence of the job losses the simplest economic model predicts, and started a long and productive debate.

Regression discontinuity

Many rules treat people differently on either side of a threshold: a scholarship for an exam score of at least 70, a benefit for households below an income line, a class split when enrolment passes 40. People just above and just below the cut-off are nearly identical in everything except the treatment, so the jump in the outcome at the cut-off estimates the effect there. This is a regression discontinuity design (RDD), introduced by the psychologists Donald Thistlethwaite and Donald Campbell in 1960.

python
import numpy as np

rng = np.random.default_rng(5)
n = 4000
score = rng.uniform(40, 100, n)                         # entrance-exam score
award = score >= 70                                     # the scholarship rule
# The first-year mark rises smoothly, and not in a straight line, with the exam score.
# The scholarship adds 3 marks on top.
mark = 25 + 60 / (1 + np.exp(-(score - 62) / 9)) + 3.0 * award + rng.normal(0, 5, n)

print(f"award holders minus the rest: {mark[award].mean() - mark[~award].mean():.2f} marks (truth 3.00)")

def side_fit(keep):
    """Least-squares line through (score - 70, mark); return its height at 70 and that height's variance."""
    X = np.column_stack([np.ones(keep.sum()), score[keep] - 70])
    coef, *_ = np.linalg.lstsq(X, mark[keep], rcond=None)
    resid = mark[keep] - X @ coef
    s2 = resid @ resid / (keep.sum() - 2)
    return coef[0], s2 * np.linalg.inv(X.T @ X)[0, 0]

print(" bandwidth   pupils used   estimated jump   standard error")
for h in (2.5, 5, 10, 20, 30):
    near = np.abs(score - 70) <= h
    (right, v1), (left, v0) = side_fit(award & near), side_fit(~award & near)
    print(f"   {h:5.1f}       {near.sum():5d}          {right - left:5.2f}          {np.sqrt(v1 + v0):.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Comparing all award holders with everyone else gives 36.32 marks, a measure of how much better strong students do, not of the scholarship. The local fits tell the real story, and it is a story about the bias–variance trade-off. A narrow window (2.5 points either side) uses only 385 students: the estimate, 2.15, is nearly unbiased but its standard error is 1.01. A window of 10 points gives 3.50 with standard error 0.54. Wide windows look precise and are wrong: with all the data, the straight lines cannot follow the curve, and the estimate of 8.48 comes with a standard error of 0.33, confidently far from the truth of 3.

40506070809010020406080100cut-off: 70entrance-exam scorefirst-year mark
The simulated scholarship data (a random sample of 140 of the 4,000 students). Each side of the cut-off has its own local straight-line fit within 10 points; the gap between them at 70, about 3.5 marks, estimates the effect of the scholarship for students near the threshold.

Two checks make an RDD credible. If people can manipulate the running variable (re-sitting until they pass, under-reporting income), a density test finds a suspicious pile-up just past the threshold. And covariates fixed before treatment, such as age, should show no jump at the cut-off. Modern practice chooses the bandwidth by a data-driven rule and reports bias-corrected confidence intervals.

History & global context

From Fisher's fields to the 2021 Nobel prize

The ideas in this lesson come from disciplines that took decades to recognise each other. Jerzy Neyman wrote down potential outcomes for agricultural plots in 1923; Ronald Fisher made randomisation the foundation of experimental design at Rothamsted; the geneticist Sewall Wright drew the first path diagrams around 1920. Epidemiology learned to argue causally from observational data in the controversy over smoking and lung cancer, with Richard Doll and Austin Bradford Hill's studies of British doctors and Hill's 1965 considerations (strength, consistency, dose–response and others) for judging whether an association is causal.

Donald Rubin turned potential outcomes into a general framework for observational studies from 1974, and with Paul Rosenbaum introduced the propensity score in 1983. Judea Pearl, a computer scientist, developed causal diagrams and the do-calculus in the 1990s. The two frameworks largely agree on what can be identified, and modern practice uses both: diagrams to decide what to adjust for, potential outcomes to define and estimate effects.

In economics, the credibility revolution of the 1990s built research around natural experiments. Joshua Angrist and Alan Krueger used children's quarter of birth, which through school-entry and leaving-age rules shifts how long they stay in school, as an instrument for the effect of schooling on earnings; Card and Krueger's minimum-wage study used a state border. In 2021 the Sveriges Riksbank Prize in Economic Sciences went half to David Card "for his empirical contributions to labour economics" and half jointly to Joshua Angrist and Guido Imbens "for their methodological contributions to the analysis of causal relationships". Alan Krueger, Card's co-author on much of that work, had died in 2019; the prize is not awarded posthumously.

ExplanationH

Causal inference in professional practice

Design before analysis. Write down the randomised trial you wish you could run (who is eligible, which treatment strategies, when follow-up starts, which outcome) and make the observational analysis imitate it. Miguel Hernán and James Robins call this target trial emulation; it prevents errors such as "immortal time bias", where the treated are, by construction, those who survived long enough to be treated. It is now standard with health records and registries, such as the Nordic national registries. Such data are about real people: access is governed by law and ethics review, and analyses run on de-identified data in secure environments.

Sensitivity analysis. Because unconfoundedness cannot be tested, careful studies ask how strong an unmeasured confounder would have to be to explain away the result. Tyler VanderWeele and Peng Ding's E-value expresses this as a single number on the risk-ratio scale: an observed risk ratio of 2 could be fully explained only by a confounder associated with both treatment and outcome by risk ratios of at least 2+2×1≈3.412 + \sqrt{2 \times 1} \approx 3.41.

Modern difference-in-differences. When units adopt a policy at different times and effects change over time, the traditional "two-way fixed effects" regression can weight comparisons in ways that make no sense, even negatively. Since about 2020, papers by Goodman-Bacon, Callaway and Sant'Anna, de Chaisemartin and D'Haultfœuille and others have given estimators that compare each cohort only with valid controls. Synthetic control, which weights untreated regions to match a single treated one before the policy, is a close relative.

Software. The methods here are a few dozen lines of NumPy, but practitioners rely on maintained packages: in R, MatchIt and packages for instruments, DiD and RDD; in Python, DoWhy, which asks you to state the causal graph and then runs refutation checks; DAGitty for diagrams and adjustment sets; and Stata, still common in economics. The R snippet below (not run here) is the shape of a typical matching analysis.

r
# R, not run here: nearest-neighbour matching on a logistic propensity score, then a balance table
library(MatchIt)
m <- matchit(subscribed ~ schooling + area, data = farms, method = "nearest", distance = "glm")
summary(m)                           # standardised mean differences before and after
matched <- match.data(m)
lm(yield ~ subscribed + schooling + area, data = matched, weights = weights)

Using machine learning to estimate propensity scores and outcome models flexibly is the subject of the frontier lesson Causal machine learning; randomised A/B tests at enormous scale are covered in the Data Science elective's Experiments in products.

Common misconceptionsH

Common misconceptions

Misconception

"Correlation never implies causation." As a warning it is wise; as a rule it is false. Under stated assumptions (randomisation, unconfoundedness given measured variables, a valid instrument, parallel trends, continuity at a threshold) associations in data do identify causal effects. The useful question is never "is it only a correlation?" but "which assumption turns this association into an effect, and how plausible is it?"

Misconception

"Controlling for more variables is always safer." Adjusting for a mediator removes part of the effect you want to measure; adjusting for a collider, or selecting the sample on one, creates bias where there was none; and adjusting for a variable that affects only the treatment (an instrument) can amplify the bias from any unmeasured confounder. What to adjust for is a causal question, answered by the diagram, not by how many columns the data set has.

Misconception

"Propensity score matching fixes confounding." It balances the covariates that went into the score, and nothing else. If an important confounder is unmeasured, matched groups remain unbalanced on it, and the estimate is biased exactly as a regression would be. Matching is a way of adjusting, not a way of discovering what to adjust for.

Misconception

"An instrumental-variable estimate is the effect of the treatment." With effects that vary between people, it is the effect for compliers, the people whose treatment the instrument changed. A lottery for school places estimates the effect for families who would take a place if offered one, which may differ from the effect of expanding the schools to everyone.

Exercises

Exercises

1. Causal inference

Warm-upHuman

A library in Kraków runs a summer reading club. With an imaginary God's-eye view, here are four children's reading-test scores in September without the club, Y(0)Y(0), and with it, Y(1)Y(1). Kofi and Lena joined; Mei and Nour did not.

Child Y(0)Y(0) Y(1)Y(1) Joined?
Kofi 30 38 yes
Lena 34 40 yes
Mei 50 52 no
Nour 46 50 no

What is the average treatment effect (ATE) of the club for these four children, in points?

2. Causal inference

CoreHuman

A study in Chile asks whether a twelve-week exercise programme (TT) lowers blood pressure (YY). The analysts believe this causal diagram:

  • age affects both joining the programme and blood pressure;
  • the programme causes weight loss, and weight loss lowers blood pressure;
  • both joining the programme and having low blood pressure make people more likely to attend the final check-up visit.

Which set of variables satisfies the backdoor criterion?

Choose one

3. Causal inference

CoreHuman

An arts-and-science academy in Nairobi admits applicants who score highly in mathematics or in music (or both). Among its students, mathematics and music scores are clearly negatively correlated, although among all applicants they are unrelated. What is the best explanation?

Choose one

4. Causal inference

CoreHuman

City P halves its bus fares in January; neighbouring city Q does not. Average weekly bus trips, in thousands:

Before After
City P (fare cut) 120 150
City Q (no change) 100 110

What is the difference-in-differences estimate of the effect of the fare cut on weekly trips in city P, in thousands?

5. Causal inference

CoreHuman

Places on a job-training course in Lyon are allocated by lottery. Among lottery winners, 50% complete the course; among losers, 10% complete it anyway (they find a place elsewhere). A year later, winners earn on average EUR 60 more per month than losers. Assuming the lottery affects earnings only through training and there are no defiers, what is the instrumental-variable (Wald) estimate of the effect of completing the course on monthly earnings, in euros?

6. Causal inference

CoreComputational

Write ipw_ate(y, t, e), which estimates the average treatment effect by normalised inverse probability weighting. The inputs are arrays (or lists) of outcomes y, treatment indicators t (0/1 or Booleans) and propensity scores e. Weight each treated unit by 1/ei1/e_i and each untreated unit by 1/(1−ei)1/(1 - e_i), compute the weighted mean outcome in each group (dividing by the sum of that group's weights), and return the difference as a float.

For example, ipw_ate([1, 2, 3, 4], [1, 1, 0, 0], [0.5, 0.25, 0.5, 0.75]) is −2.0-2.0.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Causal inference

StretchComputational

Write rd_jump(x, y, cutoff, h) for a sharp regression discontinuity. Fit a least-squares straight line to the points with cutoff <= x <= cutoff + h, and another to the points with cutoff - h <= x < cutoff. Return, as a float, the height of the right-hand line at x = cutoff minus the height of the left-hand line there.

Use NumPy only (for example np.linalg.lstsq or np.polyfit). Centring x at the cut-off makes each intercept the height you need.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources