Seeing an effect that is smaller than the noise
Halima breeds sorghum at a research station in northern Nigeria. Her new variety, she believes, yields about 0.3 tonnes per hectare more than the old one. But plots on the station differ by much more than that: the eastern fields have better soil, some plots sit in a damp hollow, birds favour the edge near the trees. A difference of 0.3 t/ha is easily swamped by differences between plots. How can she design a comparison that is fair, and precise enough to see it?
This was the problem Ronald Fisher faced when he joined Rothamsted Experimental Station in England in 1919. Rothamsted had decades of carefully recorded harvests, and Fisher found that they could not answer the questions they were meant to, because the treatments had been laid out in ways that tangled them with differences in the soil. His answer, set out in a 1926 paper and in The Design of Experiments (1935), rests on three principles: replication, randomisation and local control (blocking). A fourth, comparison against a control, was older. Together they turned the experiment into the most reliable way humans have found of learning what causes what, in fields, factories, hospitals and, today, in the software products used by billions of people.
This lesson is about designing experiments and about analysis of variance (ANOVA), the method Fisher invented to analyse them. Previous lessons asked how to analyse data that already exist. Here the statistician gets to decide how the data are made, and good design is worth more than any analysis: no method can rescue a comparison that was unfair from the start.
Why randomise?
Suppose Halima lets an experienced farmer choose which plots get the new variety. The farmer, wanting it to do well, picks the 20 plots that yielded best last year. Those plots have better soil, so the new variety's plots would out-yield the others even if the variety made no difference. The comparison is confounded: the treatment effect is mixed up with the soil effect. Randomisation, assigning the varieties by a chance mechanism, breaks the link between the treatment and everything about the plots, measured or not.
The program simulates a strip of 40 plots whose fertility rises from west to east, and a new variety whose true advantage is 0.30 t/ha. It runs 5000 seasons under three allocation schemes: the farmer's choice; a completely randomised allocation of 20 plots to each variety; and a blocked allocation, in which neighbouring plots are paired and a coin decides which of each pair gets the new variety.
import numpy as np
# 40 plots in a strip of field. Soil fertility rises from west to east, and varies plot to plot.
rng = np.random.default_rng(1926)
plots = 40
fertility = np.linspace(0, 1.2, plots) + rng.normal(0, 0.15, plots) # t/ha added by the soil
last_year = 3 + fertility + rng.normal(0, 0.3, plots) # what the farmer saw last season
effect = 0.30 # the new variety's true advantage, t/ha
def estimate(new):
"""One season: the difference in mean yield between new-variety and old-variety plots."""
y = 3 + fertility + effect * new + rng.normal(0, 0.3, plots)
return y[new == 1].mean() - y[new == 0].mean()
choice = np.zeros(plots)
choice[np.argsort(last_year)[-20:]] = 1 # the farmer picks the 20 plots that did best last year
schemes = {"farmer's choice": lambda: choice,
"completely randomised": lambda: rng.permutation(np.repeat([0.0, 1.0], 20)),
"paired blocks": lambda: np.concatenate([rng.permutation([0.0, 1.0]) for _ in range(20)])}
print(f"true effect {effect:.2f} t/ha")
for name, allocate in schemes.items():
est = np.array([estimate(allocate()) for _ in range(5000)])
print(f"{name:>22}: mean estimate {est.mean():.3f}, sd {est.std():.3f}, wrong sign {np.mean(est < 0):.1%}")Randomisation does three jobs.
- It removes systematic bias. Every plot has the same chance of each treatment, so any characteristic of the plots, known or unknown, is balanced between the groups on average.
- It makes chance the only alternative explanation. In any one randomisation the groups will differ a little by luck, but that luck is exactly what the standard error and -value measure. Without randomisation, a -value has no clear meaning.
- It justifies the test itself. Fisher's insight was that the random allocation provides the reference distribution: if the treatment did nothing, every possible allocation was equally likely to have produced the observed difference, so we can compare the observed result with all the others. That randomisation test needs no assumption of normality; we compute one below.
Control, replication and blocking
Definition— experimental unit
The experimental unit is the smallest unit to which a treatment is independently assigned: a plot, a patient, a classroom, a batch of steel. The number of experimental units, not the number of measurements, is the sample size.
Definition— control
A control group receives the standard treatment, no treatment, or a placebo (an inactive imitation), so that the treatment's effect is measured against what would otherwise have happened, rather than against nothing.
Definition— replication
Replication means applying each treatment to several independent experimental units. It is what allows the variation between units, and so the precision of a comparison, to be estimated at all.
Definition— blocking
A block is a group of experimental units expected to be similar (neighbouring plots, patients at one hospital, runs on one machine on one day). In a randomised complete block design, every treatment appears once in every block, randomised separately within each block. Differences between blocks then cancel from every treatment comparison.
The slogan is "block what you can, randomise what you cannot". Blocking removes known sources of variation from the comparisons; randomisation protects against the unknown ones. In the field, blocks are laid out across the fertility gradient, so that each block is internally uniform.
Two errors are common enough to name.
- Pseudoreplication. Measuring one plot's yield ten times, or ten plants in one pot, gives ten measurements of one experimental unit. Treating them as ten independent replicates makes the standard error far too small. The same error appears when a new teaching method is given to one class of 30 pupils and another class serves as control: the class, not the pupil, is the experimental unit, and there is one of each.
- Blinding failures. When people know which treatment a unit received, their measurements and behaviour can shift. In medicine, double-blind trials hide the allocation from both patients and the clinicians assessing them; in agriculture, the person scoring disease on leaves may be kept unaware of the plot's variety.
One-way analysis of variance
Comparing groups compared two means with a test. With more than two treatments, Halima wants first to ask one question: do the varieties differ at all? Running a test on every pair would inflate the chance of a false alarm (six pairs for four varieties). ANOVA answers the single question with one test.
With groups of sizes , total size , group means and grand mean , the total variation splits, exactly as in regression, into a part between groups and a part within them:
Theorem— The one-way ANOVA F test
If the observations are independent and normal with common variance , and all group means are equal, then
has an distribution. Large values of are evidence that the means differ.
Proof
This is the nested-model test of the previous lesson. ANOVA is a regression of on indicator variables for the groups: the fitted values are the group means, so its residual sum of squares is SSW, and the model with an intercept only has residual sum of squares SST. The statistic for dropping the indicators is , and . Under normality and the null hypothesis, SSB and SSW are independent chi-squared variables with and degrees of freedom (they are squared lengths of projections onto orthogonal subspaces), and the ratio of two independent chi-squared variables, each divided by its degrees of freedom, is -distributed.
∎
Both mean squares estimate when the null hypothesis holds; when the means differ, the between-group mean square is inflated by the spread of the true means, so tends to be large. The program analyses a completely randomised trial of four varieties with six plots each, three ways: by hand, with scipy.stats.f_oneway, and with Fisher's randomisation test, reshuffling the 24 yields among the four labels 10,000 times.
import numpy as np
from scipy import stats
# A completely randomised trial: 4 sorghum varieties, 6 plots each (synthetic yields, t/ha).
rng = np.random.default_rng(35)
names = ["A", "B", "C", "D"]
true_means = [2.8, 3.1, 3.3, 2.9]
groups = [np.round(rng.normal(m, 0.35, 6), 2) for m in true_means]
for name, g in zip(names, groups):
print(f"variety {name}: {g} mean {g.mean():.3f}")
def anova_F(groups):
y = np.concatenate(groups)
grand = y.mean()
ss_between = sum(len(g) * (g.mean() - grand) ** 2 for g in groups)
ss_within = sum(np.sum((g - g.mean()) ** 2) for g in groups)
df_b, df_w = len(groups) - 1, len(y) - len(groups)
return (ss_between / df_b) / (ss_within / df_w), ss_between, ss_within, df_b, df_w
F, ssb, ssw, df_b, df_w = anova_F(groups)
print(f"\n{'source':>9} {'SS':>7} {'df':>3} {'mean sq':>8}")
print(f"{'variety':>9} {ssb:7.3f} {df_b:3d} {ssb / df_b:8.4f}")
print(f"{'residual':>9} {ssw:7.3f} {df_w:3d} {ssw / df_w:8.4f}")
print(f"F = {F:.2f} on ({df_b}, {df_w}) df, p = {stats.f.sf(F, df_b, df_w):.4f}; R^2 = SSB/SST = {ssb / (ssb + ssw):.3f}")
check = stats.f_oneway(*groups)
print(f"scipy f_oneway: F = {check.statistic:.2f}, p = {check.pvalue:.4f}")
# Fisher's randomisation test: if variety made no difference, every reshuffle of labels was equally likely.
y = np.concatenate(groups)
perm_F = np.array([anova_F(np.split(rng.permutation(y), 4))[0] for _ in range(10_000)])
print(f"randomisation test: {np.mean(perm_F >= F):.4f} of 10,000 reshuffles gave F >= {F:.2f}")The ANOVA table gives on 3 and 20 degrees of freedom, , matching SciPy; the randomisation test, which assumes nothing about normality, gives 0.0127. Varieties account for of the variation between plots. The test says the four varieties are not all the same. It does not say which differ: variety C, at 3.40 t/ha, looks best, but deciding which pairs differ, while keeping the overall false-alarm rate controlled, needs methods such as Tukey's honestly significant difference (available as scipy.stats.tukey_hsd), discussed in multiple testing and reproducibility.
Blocks, factorials and interactions
A blocked experiment needs a blocked analysis. In a randomised complete block design with blocks and treatments, the total sum of squares splits into treatments, blocks and residual, and the treatment test uses only the residual, with degrees of freedom. Analysing the same data as if they were completely randomised leaves the block differences in the error term and can hide real effects.
The second idea is the factorial design. Rather than testing nitrogen in one experiment and irrigation in another, test every combination: low and high nitrogen, each with and without irrigation, a factorial. Fisher argued for this against the common advice of the time to vary one factor at a time, for two reasons. Every plot contributes to the estimate of both main effects, so a factorial experiment gets two experiments' worth of information for the price of one. And only a factorial can reveal an interaction: whether the effect of one factor depends on the level of the other.
The program first simulates 2000 block trials of four varieties in five blocks, analysing each with and without blocks, then analyses one factorial trial with four plots per combination.
import numpy as np
from scipy import stats
rng = np.random.default_rng(1935)
# 1. A randomised complete block design: 4 varieties in each of 5 blocks whose soils differ.
block_effect = np.array([-0.6, -0.2, 0.0, 0.3, 0.5]) # t/ha
variety_effect = np.array([0.0, 0.25, 0.40, 0.10])
def variety_p_values(y):
"""p-values for varieties, analysed with blocks (two-way) and ignoring them (one-way). Rows = blocks."""
b, v = y.shape
grand = y.mean()
ss_var = b * np.sum((y.mean(axis=0) - grand) ** 2)
ss_blk = v * np.sum((y.mean(axis=1) - grand) ** 2)
ss_res = np.sum((y - grand) ** 2) - ss_var - ss_blk
F_with = (ss_var / (v - 1)) / (ss_res / ((b - 1) * (v - 1)))
F_without = (ss_var / (v - 1)) / ((ss_res + ss_blk) / (b * v - v))
return stats.f.sf(F_with, v - 1, (b - 1) * (v - 1)), stats.f.sf(F_without, v - 1, b * v - v)
trials = np.array([variety_p_values(3 + block_effect[:, None] + variety_effect + rng.normal(0, 0.2, (5, 4)))
for _ in range(2000)])
power = np.mean(trials < 0.05, axis=0)
print(f"2000 simulated trials: varieties detected (p < 0.05) in {power[0]:.0%} with blocks, {power[1]:.0%} ignoring them")
# 2. A 2 x 2 factorial: nitrogen (low, high) x irrigation (no, yes), 4 plots per combination.
cell_true = np.array([[2.6, 3.0], # low N: no irrigation, irrigation
[3.0, 4.2]]) # high N: no irrigation, irrigation
data = cell_true[:, :, None] + rng.normal(0, 0.25, (2, 2, 4))
m = data.mean(axis=2)
print("\ncell means (rows: N low, high; columns: irrigation no, yes):")
print(np.round(m, 3))
r = data.shape[2]
grand = data.mean()
ss_N = 2 * r * np.sum((m.mean(axis=1) - grand) ** 2)
ss_I = 2 * r * np.sum((m.mean(axis=0) - grand) ** 2)
ss_NI = r * np.sum((m - m.mean(axis=1, keepdims=True) - m.mean(axis=0, keepdims=True) + grand) ** 2)
ss_E = np.sum((data - m[:, :, None]) ** 2)
df_E = 4 * (r - 1)
for name, ss in [("nitrogen", ss_N), ("irrigation", ss_I), ("N x irrigation", ss_NI)]:
F = ss / (ss_E / df_E)
print(f"{name:>15}: SS {ss:.3f}, F = {F:5.1f} on (1, {df_E}) df, p = {stats.f.sf(F, 1, df_E):.1e}")
print(f"effect of high N: {m[1, 0] - m[0, 0]:+.2f} t/ha without irrigation, {m[1, 1] - m[0, 1]:+.2f} with it")The blocked analysis detects the variety differences in 66% of simulated trials; ignoring the blocks, in 0%. The block differences (up to 1.1 t/ha) dwarf the variety differences, and left in the error term they drown them. The design was right in both analyses; only one analysis respected it.
In the factorial trial, nitrogen and irrigation both have clear main effects. The cell means tell a richer story: high nitrogen added 0.38 t/ha without irrigation and 0.77 t/ha with it. Plants short of water cannot use extra nitrogen, so the two inputs work better together than apart. On an interaction plot this shows as non-parallel lines.
Factorial designs scale up. With two-level factors, a design tests all combinations; when that is too many runs, fractional factorial designs test a carefully chosen fraction, at the cost of confounding some high-order interactions that are usually negligible. Industrial statisticians use them to tune processes with many settings, and response surface methods go on to find the best settings. In technology companies, the same logic underlies multivariate A/B tests of web pages; see the Data Science elective's experiments in products.
Randomised trials in fields and hospitals
Comparative trials are old. In 1747 the Scottish naval surgeon James Lind gave six pairs of sailors with scurvy six different remedies, and the pair given oranges and lemons recovered fastest. It was a controlled comparison, but not a randomised one, and with two sailors per treatment. Fisher's randomised field trials of the 1920s brought randomisation in; the British Medical Research Council's 1948 trial of streptomycin for pulmonary tuberculosis, designed by the statistician Austin Bradford Hill, is usually cited as the first medical trial to allocate patients by random numbers, concealed in sealed envelopes so that doctors could not steer patients into a group.
Today the randomised controlled trial (RCT) is the standard of evidence for medical treatments. Its working parts are design decisions you now recognise: a pre-registered protocol stating the primary outcome and analysis in advance (registries such as ClinicalTrials.gov); allocation concealment; blinding where possible; analysis by intention to treat, comparing patients as randomised rather than as treated, because those who stop taking a treatment differ from those who continue; a sample size justified by a power calculation; and reporting following the CONSORT guideline (collected by the EQUATOR Network). The RECOVERY trial, launched in the United Kingdom in March 2020, has randomised tens of thousands of patients admitted to hospital with COVID-19 among several treatments. Within about three months it found that the cheap steroid dexamethasone reduced deaths among patients needing oxygen or ventilation, while other widely used treatments, such as hydroxychloroquine, showed no benefit. Its design, a platform trial that adds and drops treatments under one protocol, has since been widely adopted.
Agriculture has its own long-running experiments: Rothamsted's Broadbalk wheat experiment has compared fertiliser treatments on the same strips since 1843. And since the 2000s, randomised evaluations of social programmes (cash transfers, school meals, bed-net distribution, microfinance) have reshaped development economics; Abhijit Banerjee, Esther Duflo and Michael Kremer shared the 2019 economics Nobel prize for this experimental approach, and networks such as J-PAL now run such evaluations in dozens of countries.
Honesty requires the limits too. Trials can only test what it is ethical to assign: we cannot randomise people to smoke. A trial needs equipoise, genuine uncertainty about which arm is better, informed consent, and oversight by an ethics committee, under principles such as the World Medical Association's Declaration of Helsinki. Trial populations may differ from the people who will later receive a treatment (older, sicker, from other countries), so results do not automatically transfer, a point pressed by Angus Deaton and Nancy Cartwright, who also warn that randomisation alone does not make a result true or general. Trials with small samples or many outcomes can mislead like any study. These are reasons to design trials well and read them carefully, not to dismiss them; when a question can be settled by a well-run randomised experiment, nothing else settles it as convincingly.
How many units?
An experiment too small to detect the effect it seeks wastes every unit in it, and in medicine exposes patients to risk for nothing. The power of a comparison is the probability of detecting a true effect of a given size. For two groups of units, a true difference , standard deviation , two-sided level and desired power , the normal approximation gives
The formula says what matters: grows with the square of the noise-to-signal ratio . Halving the detectable difference needs four times the units.
import numpy as np
from scipy import stats
# How many plots per variety to detect a 0.3 t/ha difference, if plot-to-plot sd is 0.5 t/ha?
delta, sigma, alpha, power = 0.3, 0.5, 0.05, 0.80
z_a, z_b = stats.norm.ppf(1 - alpha / 2), stats.norm.ppf(power)
n_formula = 2 * sigma ** 2 * (z_a + z_b) ** 2 / delta ** 2
print(f"normal-approximation formula: n = {n_formula:.1f} per group -> {int(np.ceil(n_formula))}")
# Check by simulation, with the t-test that will actually be used.
rng = np.random.default_rng(80)
for n in [20, 44, 60]:
a = rng.normal(0, sigma, (5000, n))
b = rng.normal(delta, sigma, (5000, n))
p = stats.ttest_ind(a, b, axis=1).pvalue
print(f"n = {n:2d} per group: simulated power {np.mean(p < alpha):.1%}")
# The same calculation is very sensitive to the guessed sd.
for s in [0.4, 0.5, 0.6]:
print(f"if sigma were {s}: n = {int(np.ceil(2 * s ** 2 * (z_a + z_b) ** 2 / delta ** 2))} per group")To detect Halima's 0.3 t/ha with plot-to-plot standard deviation 0.5 and 80% power at the 5% level needs 44 plots per variety. Simulating the test that will actually be used confirms it: 79.1% power at 44 per group, and only 45.9% at 20, a coin-toss experiment. The last lines show the weak point: if the standard deviation is 0.6 rather than 0.5, the answer jumps from 44 to 63. The variance guess usually comes from earlier trials or a pilot study, and good protocols state it, and its source, openly. Blocking helps here too: it reduces the effective , and since scales with , a block design that cuts the residual standard deviation by 30% roughly halves the number of plots needed.
Common misconceptions
Misconception
"Randomisation guarantees that the groups are balanced." It guarantees balance on average over possible allocations, not in any particular one. In a small experiment the groups can differ noticeably by chance; that is what the standard error accounts for, and what blocking or stratified randomisation reduces for known factors.
Misconception
"A significant ANOVA shows which groups differ." The test asks only whether all the means are equal. Identifying which pairs differ needs follow-up comparisons with a correction for multiplicity, or, better, contrasts specified before seeing the data.
Misconception
"More measurements always mean more replication." Only independent applications of a treatment count as replicates. Repeated measurements of the same plot, pupil or batch improve the measurement of that unit, but not the number of units.
Misconception
"If the main effects are significant, the interaction doesn't matter." When effects interact, a main effect is an average over conditions that may not apply to anyone. In the factorial example, "the" effect of nitrogen (about 0.57 t/ha averaged over irrigation) describes neither irrigated nor unirrigated fields well.
Exercises
1. Experimental design and analysis of variance
Warm-upHumanA school in Manila tries a new way of teaching fractions. One class of 32 pupils is taught the new way and another class of 30 the old way, by different teachers; every pupil then sits the same test. How many experimental units received each teaching method?
2. Experimental design and analysis of variance
CoreHumanA trial randomly assigns 100 patients in Lagos to a new or a standard treatment for high blood pressure. Which statement about the randomisation is correct?
3. Experimental design and analysis of variance
CoreHumanA completely randomised experiment compares 4 brands of solar lantern, with 6 lanterns of each brand, measuring hours of light per charge. The between-brand sum of squares is SSB = 12 and the within-brand sum of squares is SSW = 40. What is the statistic?
4. Experimental design and analysis of variance
CoreHumanA trial will compare two diets for dairy cows by milk yield. The standard deviation between cows is about litre per day, and the smallest difference worth detecting is litres per day. Using with and , how many cows are needed per group for 80% power at the 5% level? (Round up.)
5. Experimental design and analysis of variance
CoreHumanIn a factorial trial of a pest-control spray (no, yes) and a resistant variety (no, yes), the mean losses (%) are: no spray, ordinary variety 30; spray, ordinary variety 12; no spray, resistant variety 10; spray, resistant variety 8. Which is the best summary?
6. Experimental design and analysis of variance
CoreComputationalWrite one_way_anova(groups) that takes a list of groups (each a list or array of numbers, possibly of different sizes) and returns (F, p) for the one-way analysis of variance, computing the between- and within-group sums of squares yourself. Use scipy.stats.f.sf for the -value.
7. Experimental design and analysis of variance
StretchComputationalWrite rcbd_anova(y) for a randomised complete block design. y is a 2-D array whose rows are blocks and columns are treatments, one observation per cell. Return (F, p) for the treatment effect, using
- and ,
- with degrees of freedom,
- .
The tests check your against an independent calculation: the nested-model test from a regression with block and treatment indicators.
Further reading and resources
- Rothamsted ResearchWhere Fisher developed experimental design; home of the Broadbalk wheat experiment, running since 1843.
- The RECOVERY trialA large randomised platform trial of treatments for patients admitted to hospital with COVID-19, with its protocol and results.
- EQUATOR NetworkReporting guidelines for health research, including CONSORT for randomised trials.
- ClinicalTrials.govA public registry where trials are registered before they start, so that unfavourable results cannot quietly disappear.
- The Abdul Latif Jameel Poverty Action Lab (J-PAL)Randomised evaluations of social programmes in many countries.
- World Medical Association: Declaration of HelsinkiThe ethical principles for medical research involving people.
- The Design of Experiments · R. A. Fisher · 1935The book that set out randomisation, replication and blocking, and the "lady tasting tea" experiment.
- Understanding and misunderstanding randomized controlled trials · Angus Deaton and Nancy Cartwright · 2018Social Science & Medicine, volume 210. A careful critique of what trials can and cannot show.
- NIST/SEMATECH e-Handbook: process improvement (design of experiments) · National Institute of Standards and Technology