Who benefits, and by how much?
Causal inference asked whether a programme works on average, and showed how randomisation, adjustment and natural experiments answer that. Practice keeps asking two harder questions.
The first is about adjustment with many covariates. Suppose an agricultural ministry wants to know whether a drought-tolerant seed raises yields, using survey data on thousands of farms. Adopters differ from non-adopters in rainfall, soil, farm size, distance to market and a dozen other things, and those variables affect yield in curved, interacting ways. A linear regression with these covariates as controls is easy to run and quietly wrong when the true relationships are not linear. A random forest or boosted trees could learn the relationships — but how do you get a trustworthy confidence interval for a causal effect out of a black box?
The second is about heterogeneity. Suppose a district health service is trialling text-message reminders for vaccination appointments. The average effect is a few percentage points, but the service can only afford to send reminders to some families. For whom does a reminder make the difference? A seed may help on sandy soil and not on clay; a drug may help those at high risk and harm those at low risk.
Causal machine learning is the body of methods, mostly developed since about 2015 by statisticians and econometricians, that uses flexible predictive models for both jobs while keeping the guarantees of classical inference: estimates that converge at the usual rate, and confidence intervals that mean what they say. Its central idea is simple to state. Machine learning is used only for the parts of the problem that are prediction — how covariates relate to treatment and to outcome — and those predictions are combined in a way that makes the causal estimate insensitive to their errors.
Targets and assumptions
We keep the notation of Causal inference: a binary treatment , potential outcomes and , covariates , and the propensity score .
Definition— Conditional average treatment effect (CATE)
The CATE is : the average effect among units with covariates . The average treatment effect is its average, .
The CATE is not an individual effect. Two farms with identical recorded covariates can respond differently; is the average over all such farms, and the finer the covariates, the closer it gets to — but never reaches — the individual. Identification needs the familiar assumptions: unconfoundedness ( independent of given ), overlap (), and SUTVA (no interference, one version of treatment). Under them,
and the ATE can be written in several equivalent ways using , , and . These regression functions are called nuisance functions: we do not care about them for their own sake, but we need them to get at the effect, and estimating them is a prediction problem.
A convenient special case is the partially linear model:
The effect is constant, while (how covariates drive the outcome) and (how they drive treatment; for binary , ) can be arbitrarily complicated. When the effect does vary, the estimated in this model is a weighted average of , with the most weight where treatment is most uncertain.
Why not just plug in machine learning?
The obvious approach is to fit a flexible model for given and read off the effect of : predict every unit's outcome with set to 1 and to 0, and average the difference (the S-learner). Or fit with a flexible model and regress what is left on . Both approaches inherit two biases from the learner.
Regularisation bias. Flexible learners avoid overfitting by shrinking, pruning or stopping early, so they estimate with a bias that disappears only slowly as grows — more slowly than . Whatever part of the learner misses, and that is correlated with through , is picked up by the treatment coefficient. The estimate of then converges at the learner's slow rate, and a confidence interval of width around it excludes the truth more and more often as the sample grows.
Overfitting bias. An adaptive learner fitted and evaluated on the same observations partly memorises each one's noise, and the memorised noise contaminates the residuals that the effect is estimated from.
The repair for the first bias is an old idea from econometrics. Peter Robinson showed in 1988 that in the partially linear model you can partial out from both sides:
so regressing the outcome residual on the treatment residual estimates . This is the Frisch–Waugh–Lovell theorem of multiple regression with the linear projections replaced by arbitrary regression functions. Its advantage is that errors in and enter the estimate only through their product, so moderate errors in each become a small error in . The repair for the second bias is cross-fitting: never compute a unit's residual with a model that was trained on that unit.
Double/debiased machine learning
Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey and Robins turned these two repairs into a general recipe.
Definition— Neyman orthogonality
Suppose solves a moment equation , where is one observation and the nuisance functions. The score is Neyman orthogonal if the moment is locally insensitive to the nuisances: the derivative of with respect to , at , is zero for every direction . For the partially linear model, is orthogonal; the naive score is not.
Definition— Cross-fitting
Split the sample into folds. For each fold , fit the nuisance models on the other folds and evaluate them on fold . Pool the out-of-fold predictions from all folds and solve the moment equation once on the full sample.
Theorem— Double/debiased machine learning (informal)
With a Neyman-orthogonal score and cross-fitting, suppose overlap holds and the nuisance estimators are consistent with errors satisfying — for example, each error shrinking faster than . Then is asymptotically normal with the same variance as if the nuisances were known, and the usual sandwich standard error gives confidence intervals with correct coverage. For the partially linear model the variance is .
The rate condition is the honest fine print. is far slower than the parametric , and lasso, random forests, boosting and neural networks achieve it under various smoothness or sparsity assumptions — but not always, and not for any learner on any data. "Double" refers to the two nuisance regressions (and to the earlier "double selection" lasso method of Belloni, Chernozhukov and Hansen); "debiased", to removing the regularisation bias.
DML on synthetic farms
The code simulates 4,000 farms. Adoption of the new seed depends on rainfall, soil and other covariates through curved functions; so does yield. The true effect is 1 tonne per hectare. The flexible learner is deliberately simple — ridge regression on a set of bump functions of each covariate, an additive smoother you could replace with a random forest or boosted trees — so that the whole method fits on one screen.
import numpy as np
rng = np.random.default_rng(2024)
n, theta = 4000, 1.0 # farms, and the true effect of the new seed (t/ha)
X = rng.normal(size=(n, 6)) # standardised covariates: rainfall, soil quality, farm size, ...
g = 2 * np.sin(1.5 * X[:, 0]) + 0.8 * X[:, 1] ** 2 + 0.5 * X[:, 2] # yield without the seed
e = 1 / (1 + np.exp(-(np.sin(1.5 * X[:, 0]) + 0.4 * (X[:, 1] ** 2 - 1) + 0.5 * X[:, 2])))
T = (rng.random(n) < e).astype(float) # adoption depends on the same covariates: confounding
Y = theta * T + g + rng.normal(0, 1, n)
centres = np.linspace(-2.5, 2.5, 11)
def learner(X_tr, y_tr, X_new, lam=1.0):
"""A flexible regression: ridge on 11 Gaussian bumps per covariate (an additive smoother)."""
def basis(Z):
bumps = np.exp(-((Z[:, :, None] - centres) ** 2) / (2 * 0.5**2))
return np.column_stack([Z, bumps.reshape(len(Z), -1)])
F, G = basis(X_tr), basis(X_new)
mu_f, mu_y = F.mean(axis=0), y_tr.mean()
coef = np.linalg.solve((F - mu_f).T @ (F - mu_f) + lam * np.eye(F.shape[1]), (F - mu_f).T @ (y_tr - mu_y))
return (G - mu_f) @ coef + mu_y
def dml_plr(Y, T, X, n_folds=5, seed=0):
"""Double/debiased ML for Y = theta*T + g(X) + noise. Cross-fitting: each unit's residuals
come from models trained on the other folds, never on the unit itself."""
folds = np.random.default_rng(seed).permutation(len(Y)) % n_folds
y_res, t_res = np.empty(len(Y)), np.empty(len(Y))
for k in range(n_folds):
tr, te = folds != k, folds == k
y_res[te] = Y[te] - learner(X[tr], Y[tr], X[te]) # Y minus an estimate of E[Y | X]
t_res[te] = T[te] - learner(X[tr], T[tr], X[te]) # T minus an estimate of E[T | X]
est = (t_res @ y_res) / (t_res @ t_res) # regress residual on residual
eps = y_res - est * t_res
se = np.sqrt(np.mean(t_res**2 * eps**2) / np.mean(t_res**2) ** 2 / len(Y))
return est, se, y_res, t_res
def ols(design, y):
beta, *_ = np.linalg.lstsq(design, y, rcond=None)
cov = np.linalg.inv(design.T @ design) * np.var(y - design @ beta)
return beta[1], np.sqrt(cov[1, 1])
ones = np.ones(n)
est, se, y_res, t_res = dml_plr(Y, T, X)
print(f"true effect {theta:.2f} t/ha")
for name, (b, s) in [("difference in means", ols(np.column_stack([ones, T]), Y)),
("OLS, linear controls", ols(np.column_stack([ones, T, X]), Y)),
("DML, cross-fitted", (est, se))]:
print(f"{name:22s} {b:6.3f} 95% CI [{b - 1.96 * s:.3f}, {b + 1.96 * s:.3f}]")
# Sensitivity (Cinelli and Hazlett's robustness value, applied to the residualised regression):
r2 = np.corrcoef(t_res, y_res)[0, 1] ** 2 # partial R^2 of T with Y, given X
f2 = r2 / (1 - r2)
rv = 0.5 * (np.sqrt(f2**2 + 4 * f2) - f2)
print(f"\npartial R^2 of treatment with outcome, given X: {r2:.3f}")
print(f"robustness value: {rv:.3f} (a hidden confounder explaining {100 * rv:.0f}% of the residual variance")
print(f"of both treatment and outcome would be needed to bring the estimate to zero)")The last lines of output are a sensitivity analysis, which we return to below: given everything we measured, an unmeasured confounder would have to explain about 36% of the remaining variance of both adoption and yield to explain the whole effect away.
One run proves little. The real question about a confidence interval is how often it covers the truth, and that takes a simulation study. Below, 200 synthetic studies of 1,000 farms each are analysed three ways; the third uses the same learner with a penalty 300 times stronger, so its nuisance estimates are badly over-smoothed.
import numpy as np
rng = np.random.default_rng(7)
theta, n, reps = 1.0, 1000, 200
centres = np.linspace(-2.5, 2.5, 11)
def simulate(n):
X = rng.normal(size=(n, 6))
g = 2 * np.sin(1.5 * X[:, 0]) + 0.8 * X[:, 1] ** 2 + 0.5 * X[:, 2]
e = 1 / (1 + np.exp(-(np.sin(1.5 * X[:, 0]) + 0.4 * (X[:, 1] ** 2 - 1) + 0.5 * X[:, 2])))
T = (rng.random(n) < e).astype(float)
return X, T, theta * T + g + rng.normal(0, 1, n), e, theta * e + g # last two: true E[T|X], E[Y|X]
def basis(Z): # the same additive bumps as before, computed once per dataset
bumps = np.exp(-((Z[:, :, None] - centres) ** 2) / (2 * 0.5**2))
return np.column_stack([Z, bumps.reshape(len(Z), -1)])
def ridge(F_tr, y_tr, F_new, lam):
mu_f, mu_y = F_tr.mean(axis=0), y_tr.mean()
coef = np.linalg.solve((F_tr - mu_f).T @ (F_tr - mu_f) + lam * np.eye(F_tr.shape[1]), (F_tr - mu_f).T @ (y_tr - mu_y))
return (F_new - mu_f) @ coef + mu_y
def dml_plr(Y, T, F, lam, n_folds=5):
folds = np.arange(len(Y)) % n_folds
y_res, t_res = np.empty(len(Y)), np.empty(len(Y))
for k in range(n_folds):
tr, te = folds != k, folds == k
y_res[te] = Y[te] - ridge(F[tr], Y[tr], F[te], lam)
t_res[te] = T[te] - ridge(F[tr], T[tr], F[te], lam)
est = (t_res @ y_res) / (t_res @ t_res)
eps = y_res - est * t_res
se = np.sqrt(np.mean(t_res**2 * eps**2) / np.mean(t_res**2) ** 2 / len(Y))
return est, se, T - t_res, Y - y_res # also return the fitted nuisances
def ols_linear(Y, T, X):
Z = np.column_stack([np.ones(len(Y)), T, X])
beta, *_ = np.linalg.lstsq(Z, Y, rcond=None)
cov = np.linalg.inv(Z.T @ Z) * np.var(Y - Z @ beta)
return beta[1], np.sqrt(cov[1, 1])
rows = {"OLS, linear controls": [], "DML, lambda = 1": [], "DML, lambda = 300": []}
errors = {"DML, lambda = 1": [], "DML, lambda = 300": []}
for _ in range(reps):
X, T, Y, e, ell = simulate(n)
F = basis(X)
rows["OLS, linear controls"].append(ols_linear(Y, T, X))
for lam in (1, 300):
est, se, e_hat, ell_hat = dml_plr(Y, T, F, lam)
rows[f"DML, lambda = {lam}"].append((est, se))
errors[f"DML, lambda = {lam}"].append([np.sqrt(np.mean((e_hat - e) ** 2)), np.sqrt(np.mean((ell_hat - ell) ** 2))])
print(f"{reps} simulated studies of {n} farms each; true effect {theta}")
print(f"{'estimator':22s} {'bias':>7} {'sd':>6} {'mean SE':>8} {'CI covers':>10} {'RMSE of E[T|X]':>15} {'of E[Y|X]':>10}")
for name, r in rows.items():
r = np.array(r)
cover = np.mean(np.abs(r[:, 0] - theta) <= 1.96 * r[:, 1])
err = "".join(f"{v:{w}.3f}" for v, w in zip(np.mean(errors[name], axis=0), (15, 11))) if name in errors else ""
print(f"{name:22s} {r[:, 0].mean() - theta:+7.3f} {r[:, 0].std():6.3f} {r[:, 1].mean():8.3f} {cover:10.2f} {err}")With a reasonable learner, DML is essentially unbiased (+0.007) and its nominal 95% intervals cover the truth in 94% of studies, while linear adjustment never does. The over-regularised learner shows the limit of the method: its estimate of is three times as wrong (root-mean-square error 1.17 against 0.36), the product of the nuisance errors no longer vanishes fast enough, and the bias, 0.49, is five standard errors. Orthogonality forgives moderate errors in the nuisances; it cannot rescue bad ones. In practice that means tuning the learners by cross-validated prediction error, reporting how well they predict, and checking that results are stable across several reasonable learners.
Heterogeneous effects
Estimating is harder than estimating an average, for a reason worth saying plainly: there is no column of true effects to train on. We never see for anyone, so we cannot simply fit a model to it and check its test error. The meta-learners are different ways of building a CATE estimator from ordinary predictive models (Künzel et al.):
- The S-learner fits one model of on and differences its predictions. Regularisation tends to shrink the effect of towards zero.
- The T-learner fits separate models and in each arm and subtracts them. Each model is regularised for its own prediction task, not for their difference, so the difference can be noisy and biased, especially when one arm is small.
- The X-learner imputes individual effects using the other arm's model and then smooths them, which helps with unbalanced arms.
- The R-learner (Nie and Wager) estimates by minimising plus a penalty: Robinson's partialling out with replaced by a function.
- The DR-learner (Kennedy) builds, by cross-fitting, a doubly robust pseudo-outcome
whose conditional mean is approximately , and regresses it on . The same , averaged, is the augmented inverse-probability-weighted (AIPW) estimator of the ATE — DML for the "interactive" model in which effects may vary.
The simulation below is a synthetic randomised trial of vaccination reminders for 40,000 families, with a second trial of the same size held out for evaluation. Reminders help most when a covariate (think of it as distance to the clinic) is large, and for some families they slightly lower attendance.
import numpy as np
rng = np.random.default_rng(5)
sigmoid = lambda z: 1 / (1 + np.exp(-z))
def simulate(n):
"""A randomised SMS-reminder trial (synthetic). Y = 1 if the appointment is kept."""
X = rng.normal(size=(n, 5)) # standardised: past attendance, distance to clinic, ...
p0 = sigmoid(0.2 + 1.5 * X[:, 0]) # chance of attending without a reminder
p1 = np.clip(p0 + 0.15 * sigmoid(3 * X[:, 1]) - 0.04, 0, 1) # with one: helps most when X[:, 1] is large
tau = p1 - p0 # the true conditional effect at X (never observed)
T = rng.integers(0, 2, n) # randomised: half get the reminder
Y = (rng.random(n) < np.where(T == 1, p1, p0)).astype(float)
return X, T, Y, tau, p0
centres = np.linspace(-2.5, 2.5, 9)
def smoother(X_tr, y_tr, X_new, lam=3.0):
def basis(Z):
return np.column_stack([Z, np.exp(-((Z[:, :, None] - centres) ** 2) / 0.5).reshape(len(Z), -1)])
F, G = basis(X_tr), basis(X_new)
mu_f, mu_y = F.mean(axis=0), y_tr.mean()
coef = np.linalg.solve((F - mu_f).T @ (F - mu_f) + lam * np.eye(F.shape[1]), (F - mu_f).T @ (y_tr - mu_y))
return (G - mu_f) @ coef + mu_y
X, T, Y, tau, p0 = simulate(40000) # training data from the trial
Xt, Tt, Yt, tau_t, p0_t = simulate(40000) # a second, held-out trial for evaluation
# T-learner: one model per arm, subtract.
mu1 = lambda Z: smoother(X[T == 1], Y[T == 1], Z)
mu0 = lambda Z: smoother(X[T == 0], Y[T == 0], Z)
cate_T = mu1(Xt) - mu0(Xt)
# DR-learner: cross-fitted doubly robust pseudo-outcomes (propensity 0.5 is known), then smooth them.
folds = np.arange(len(Y)) % 2
psi = np.empty(len(Y))
for k in (0, 1):
tr, te = folds != k, folds == k
m1 = smoother(X[tr & (T == 1)], Y[tr & (T == 1)], X[te])
m0 = smoother(X[tr & (T == 0)], Y[tr & (T == 0)], X[te])
psi[te] = m1 - m0 + T[te] * (Y[te] - m1) / 0.5 - (1 - T[te]) * (Y[te] - m0) / 0.5
cate_DR = smoother(X, psi, Xt, lam=30.0)
print(f"true average effect {tau_t.mean():.4f}; effects range {tau_t.min():.3f} to {tau_t.max():.3f}")
for name, c in [("T-learner", cate_T), ("DR-learner", cate_DR)]:
print(f"{name:10s} RMSE vs true effect {np.sqrt(np.mean((c - tau_t) ** 2)):.4f}, correlation {np.corrcoef(c, tau_t)[0, 1]:.2f}")
# Best linear projection of the effect on distance (X[:, 1]): a heterogeneity summary with a valid CI.
Z = np.column_stack([np.ones(len(Y)), X[:, 1]])
b = np.linalg.solve(Z.T @ Z, Z.T @ psi)
r = psi - Z @ b
V = np.linalg.inv(Z.T @ Z) @ (Z.T * r**2) @ Z @ np.linalg.inv(Z.T @ Z) # heteroscedasticity-robust
print(f"\nbest linear projection: effect = {b[0]:.4f} + {b[1]:.4f} x distance (slope SE {np.sqrt(V[1, 1]):.4f})")
# Uplift evaluation on the held-out trial: whom should 20% of the reminders go to?
def incremental(score, frac):
top = np.argsort(-score)[: int(frac * len(score))]
y1, y0 = Yt[top][Tt[top] == 1], Yt[top][Tt[top] == 0]
est = (y1.mean() - y0.mean()) * len(top) # extra appointments kept, estimated from the trial
se = np.sqrt(y1.var() / len(y1) + y0.var() / len(y0)) * len(top)
return est, se
print(f"\nextra appointments kept by reminding 20% of {len(Yt)} people, estimated from the held-out trial:")
for name, s in [("true effect (oracle)", tau_t), ("DR-learner", cate_DR), ("T-learner", cate_T),
("most likely to attend", mu0(Xt)), ("least likely to attend", -mu0(Xt)),
("random", rng.random(len(Yt)))]:
est, se = incremental(s, 0.2)
print(f" ranked by {name:24s} {est:6.0f} +/- {1.96 * se:.0f}")Both learners recover the shape of the effect well (correlation about 0.9 with the truth), and in a randomised trial with a known propensity of 0.5 the DR-learner's advantage is modest. It grows when treatment is unbalanced or confounded, where the T-learner's two separately regularised models disagree for reasons that have nothing to do with the effect.
Two outputs deserve attention. The best linear projection summarises heterogeneity in one interpretable line — effect distance, slope standard error 0.004 — and, because it is a regression of a cross-fitted orthogonal pseudo-outcome, its confidence interval is valid even though the nuisances came from machine learning (Semenova and Chernozhukov). Individual CATE estimates, by contrast, come with no such guarantee. And the targeting evaluation at the end uses only the held-out trial: reminding the top 20% ranked by the DR-learner yields an estimated 716 extra appointments kept (±174), close to the 756 achieved by ranking on the true effects, and about three times the yield of random targeting.
Causal forests and uplift modelling
Causal forests. A regression tree splits the data to make outcomes homogeneous within leaves. A causal tree splits to make treatment effects differ between leaves, which needs a criterion based on estimated effects rather than outcomes. Athey and Imbens added honesty: one half of the sample chooses the splits and the other half estimates the effects within the leaves, so that the same noise cannot both create a split and fill it. Wager and Athey averaged many honest trees into a causal forest and proved that its estimates are asymptotically normal, with pointwise confidence intervals; generalised random forests (Athey, Tibshirani and Wager) recast the forest as an adaptive way of finding neighbours and then solve a local moment equation, with orthogonalised (partialled-out) inputs. The reference implementation is the R package grf; a typical analysis (not run here) looks like this:
# R, not run here: a causal forest with the grf package (W is the treatment indicator)
library(grf)
cf <- causal_forest(X, Y, W, num.trees = 2000)
average_treatment_effect(cf, target.sample = "all") # doubly robust ATE with standard error
best_linear_projection(cf, X[, c("distance", "past_attendance")])
test_calibration(cf) # is there detectable heterogeneity at all?
tau_hat <- predict(cf)$predictions # out-of-bag CATE estimatesPython users have EconML and CausalML, and DoubleML implements DML in both languages; they differ more in interface and defaults than in the underlying ideas.
Uplift modelling is the name the marketing and public-health communities gave to CATE estimation for targeting, usually with a binary outcome and data from a randomised campaign. Its vocabulary is useful anywhere: people who act only if contacted (persuadables), who act anyway (sure things), who never act (lost causes), and who are put off by contact (sleeping dogs). A traditional response model targets those most likely to act, which spends the budget on sure things. In the simulation, ranking by "most likely to attend" produced 245 extra appointments, no better than random — while ranking by estimated uplift produced nearly three times as many. Uplift models are evaluated with uplift (Qini) curves like the targeting table above, computed on held-out randomised data; Yadlowsky et al. put such curves on a firm inferential footing with the rank-weighted average treatment effect (RATE). The Data Science elective's Experiments in products covers the randomised experiments that supply such data.
Sensitivity analysis
Every method in this lesson assumes unconfoundedness, and none can test it. The flexibility of machine learning makes the assumption more tempting to forget, not less: a model with two hundred covariates feels as if it must have caught every confounder.
Causal inference introduced the E-value. For regression-type estimates, Carlos Cinelli and Chad Hazlett (whose sensemakr software implements the ideas) express the omitted-variable bias in terms of two partial values: how much of the residual variance of the treatment, and of the outcome, an unmeasured confounder would explain. Their robustness value is the strength — equal on both sides — at which would bring the estimate to zero,
and it can be compared with the explanatory power of the strongest measured covariates ("would have to be three times as strong as rainfall?"). In the farm example the robustness value was 0.365. Chernozhukov, Cinelli, Newey, Sharma and Syrgkanis extended these bounds to DML with nonlinear nuisances. None of this proves an effect exists. It turns an unanswerable question ("is there an unmeasured confounder?") into an arguable one ("is there one this strong?") that subject experts can debate.
Where these methods are used, and misused (2026)
Uses. Economists use DML and causal forests to evaluate labour-market programmes, cash transfers and tax policies from administrative data, and to study heterogeneity in randomised field experiments; one widely cited example is the analysis of immunisation incentives in India by Chernozhukov, Demirer, Duflo and Fernández-Val. Technology companies use uplift models and CATE estimates from their A/B tests to decide who receives a notification, a discount or a new feature. Medical researchers use these methods to explore heterogeneity of treatment effects in trials and in registry data, and to emulate target trials with flexible adjustment.
Misuses, all common:
- Treating flexibility as a cure for confounding. DML removes modelling bias given the right covariates. It does nothing about missing covariates, and it is only as unconfounded as the data.
- Bad controls. Feeding every available variable into the nuisance models includes post-treatment variables (mediators, outcomes of treatment) and colliders, which create bias rather than remove it. Covariates must be chosen with a causal diagram, measured before treatment.
- Ignoring overlap. Where is near 0 or 1, pseudo-outcomes divide by tiny numbers and explode. Trimming such units is often sensible, but it changes the population the estimate describes, and that must be said.
- Fishing for heterogeneity. Search enough subgroups and some will show large effects by chance (Multiple testing and reproducibility). Pre-specify the main heterogeneity analysis, test for heterogeneity as a whole (calibration tests, RATE) before interpreting individual estimates, and validate targeting rules on held-out randomised data. Clinical guidance on predictive approaches to treatment-effect heterogeneity makes the same demands.
- Reading CATE as individual destiny. is a noisy estimate of an average for people like . Deciding who gets a medical treatment, a loan or a public service from it raises fairness and legal questions — targeting by estimated benefit can encode existing inequalities in the data — that the estimator cannot answer.
- Mistaking variable importance for mechanism. A covariate the forest splits on often may be a proxy for something else entirely.
Common misconceptions
Misconception
"With enough covariates and a powerful model, the estimate is causal." Causality comes from the assumptions — unconfoundedness, overlap, no interference — not from the model. A perfect predictor of from and still confounds if an unmeasured drives both.
Misconception
"DML works with any learner." It needs nuisance estimates that are good enough — errors whose product shrinks faster than . The over-regularised learner in the simulation produced a bias of five standard errors and intervals that never covered the truth.
Misconception
"I can check a CATE model by its test error, like any other model." The true effects are never observed, so there is no test error to compute. CATE models are judged indirectly: by the targeting gains they deliver on held-out randomised data, by calibration tests, and by simulations where the truth is known.
Misconception
"Targeting the people most likely to respond is uplift modelling." That is a response model, and it wastes effort on people who would respond anyway. Uplift targets the difference that treatment makes, which needs randomised (or credibly unconfounded) data with both treated and untreated units.
Exercises
1. Causal machine learning
Warm-upHumanIn the farm simulation, OLS with all six covariates as linear controls estimated the effect as 1.97 t/ha, with a 95% interval far from the true 1.0. What is the main reason?
2. Causal machine learning
CoreHumanA DML analysis finds that the partial of the treatment with the outcome, given the covariates, is . Using and , what is the robustness value, as a percentage rounded to the nearest whole number?
3. Causal machine learning
CoreHumanWhich of the following are needed for a DML confidence interval for a treatment effect to be trustworthy? Select all that apply.
4. Causal machine learning
CoreComputationalWrite crossfit_residuals(X, v, learner, n_folds) returning the cross-fitted residuals of v (a 1-D array) given X (2-D). Assign row i to fold i % n_folds. For each fold, call learner(X_train, v_train, X_fold) with the rows of the other folds as training data, and set the residuals of the fold's rows to v_fold - predictions. Return an array the same length as v.
5. Causal machine learning
StretchComputationalWrite aipw_ate(Y, T, mu1, mu0, e) returning (estimate, se) for the average treatment effect, using the doubly robust (AIPW) pseudo-outcome
The estimate is the mean of , and se is its standard deviation divided by . The arguments mu1, mu0 and e are arrays of predictions at each unit. The tests check double robustness: the estimate stays close to the truth when the outcome models are badly wrong but the propensity is right.
6. Causal machine learning
CoreHumanA clinic can send 1,000 reminders. Its data scientist proposes sending them to the 1,000 patients with the highest predicted probability of attending, from a model trained on past attendance. What is the main problem?
Further reading and resources
- Double/Debiased Machine Learning for Treatment and Causal Parameters · Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey and James Robins · 2016The paper that set out Neyman orthogonality and cross-fitting for machine-learning nuisances (published in The Econometrics Journal, 2018).
- Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning · Sören Künzel, Jasjeet Sekhon, Peter Bickel and Bin Yu · 2017
- Quasi-Oracle Estimation of Heterogeneous Treatment Effects · Xinkun Nie and Stefan Wager · 2017The R-learner.
- Towards optimal doubly robust estimation of heterogeneous causal effects · Edward Kennedy · 2020The DR-learner and its guarantees.
- Recursive Partitioning for Heterogeneous Causal Effects · Susan Athey and Guido Imbens · 2015Honest causal trees.
- Estimation and Inference of Heterogeneous Treatment Effects using Random Forests · Stefan Wager and Susan Athey · 2015
- Generalized Random Forests · Susan Athey, Julie Tibshirani and Stefan Wager · 2016
- Debiased Machine Learning of Conditional Average Treatment Effects and Other Causal Functions · Vira Semenova and Victor Chernozhukov · 2017Best linear projections of the CATE with valid inference.
- Generic Machine Learning Inference on Heterogenous Treatment Effects in Randomized Experiments · Victor Chernozhukov, Mert Demirer, Esther Duflo and Iván Fernández-Val · 2017Inference on heterogeneity in randomised trials, with an application to immunisation incentives in India.
- Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects · Steve Yadlowsky, Scott Fleming, Nigam Shah, Emma Brunskill and Stefan Wager · 2021
- Inference on Treatment Effects After Selection Amongst High-Dimensional Controls · Alexandre Belloni, Victor Chernozhukov and Christian Hansen · 2012Double selection with the lasso, a precursor of DML.
- Long Story Short: Omitted Variable Bias in Causal Machine Learning · Victor Chernozhukov, Carlos Cinelli, Whitney Newey, Amit Sharma and Vasilis Syrgkanis · 2021
- Making sense of sensitivity: extending omitted variable bias · Carlos Cinelli and Chad Hazlett · 2020Journal of the Royal Statistical Society, Series B. The robustness value used in this lesson.
- grf (generalized random forests)The reference R implementation of causal forests.
- DoubleMLDouble/debiased machine learning in Python and R.
- EconMLA Python library of heterogeneous-effect estimators.
- CausalMLA Python library focused on uplift modelling and meta-learners.
- sensemakrSensitivity analysis to unobserved confounding, in R.