Is the difference real?
Most questions that reach a statistician are comparisons. Does an improved sorghum variety yield more than the local one? Do runners go faster in a new shoe? Do SMS reminders bring more children back for their second vaccine dose? Do patients stay longer on one hospital ward than another? In each case two groups differ in the data, and the question is whether the difference reflects something real or just the luck of who ended up in which group.
Hypothesis testing gave the logic: assume no real difference, ask how surprising the data would be, and report the size of the difference with its interval. This lesson supplies the tools for the commonest comparisons. There are many of them, and textbooks often present them as a menu. That invites the worst habit in applied statistics: running several tests and reporting the one that gives the smallest p-value. The better way is to let the design of the study and the question being asked choose the test, before the data are seen. By the end of the lesson you will have a short decision guide, and a simulation showing what shopping among tests costs.
Signal, noise, and which noise counts
Every comparison has the same skeleton: a signal (the difference between the groups) divided by the noise against which it must be judged (the standard error of that difference). What changes from design to design is which noise counts.
Consider twelve runners who each run 5 km twice, once in shoe A and once in shoe B. Runners differ enormously: the fastest finish in about 20 minutes, the slowest in nearly 28. The shoe makes a difference of about 12 seconds. If we throw the 24 times into two piles labelled A and B, the 12-second signal is buried under minutes of differences between runners. But each runner wore both shoes, so we can compare each runner with themselves. The differences between runners cancel, and the shoe effect stands out.
This is why the first question is always how the groups were formed. In independent groups, different units (plots, people, patients) are in each group, and the noise includes all the variation between units. In a paired (or matched) design, each unit appears in both conditions, or units are deliberately matched in pairs, and the noise is only the variation within pairs. The second question is what kind of outcome was measured: a number (yield, time, length of stay) or a category (returned or not, type of transport).
Definitions
Definition— Welch's two-sample t-test
For independent samples of sizes with means and standard deviations , Welch's statistic is compared with a distribution whose degrees of freedom are estimated by the Welch–Satterthwaite formula Student's pooled test instead combines the two variances into one, which is valid only if the populations have equal variances.
Definition— paired t-test
For pairs , compute the differences and apply the one-sample -test to them: with degrees of freedom.
Definition— chi-squared test of independence
For a table of counts with rows and columns, the expected count in each cell under the null hypothesis of no association is (row total × column total) / grand total. Pearson's chi-squared statistic is compared with a chi-squared distribution with degrees of freedom. For a 2×2 table it equals the square of the two-proportion statistic.
Definition— Mann–Whitney test
The Mann–Whitney (or Wilcoxon rank-sum) test pools the two samples, ranks all the values, and asks whether one group's ranks are systematically higher. It tests whether , the chance that a random member of one group exceeds a random member of the other, equals .
Definition— permutation test
A permutation test computes a statistic (such as the difference in means), then recomputes it after randomly reshuffling the group labels many times. The p-value is the share of reshuffles giving a statistic at least as extreme as the observed one. It is valid when, under the null hypothesis, the labels are exchangeable: any assignment of labels to the observed values was equally likely, as it is in a randomised experiment where the treatment truly does nothing.
Two independent groups: Welch's t-test
An agricultural research station compares a local sorghum variety on 12 plots with an improved variety on 15 plots (synthetic data). We compute Welch's test by hand and with SciPy, then ask how Student's pooled test behaves when the two groups have different spreads.
import numpy as np
from scipy import stats
# Synthetic sorghum yields (t/ha) from plots of two varieties at a research station.
rng = np.random.default_rng(51)
local = np.round(rng.normal(2.9, 0.35, 12), 2)
improved = np.round(rng.normal(3.3, 0.6, 15), 2)
def welch(a, b):
"""Welch's t statistic and its Welch-Satterthwaite degrees of freedom."""
va, vb = a.var(ddof=1) / a.size, b.var(ddof=1) / b.size
t = (b.mean() - a.mean()) / np.sqrt(va + vb)
df = (va + vb) ** 2 / (va**2 / (a.size - 1) + vb**2 / (b.size - 1))
return t, df
t, df = welch(local, improved)
p = 2 * stats.t.sf(abs(t), df)
print(f"local mean {local.mean():.3f} (sd {local.std(ddof=1):.3f}), improved mean {improved.mean():.3f} (sd {improved.std(ddof=1):.3f})")
print(f"by hand: t = {t:.3f}, df = {df:.1f}, p = {p:.4f}")
res = stats.ttest_ind(improved, local, equal_var=False)
ci = res.confidence_interval(0.95)
print(f"SciPy: t = {res.statistic:.3f}, p = {res.pvalue:.4f}, 95% CI for the difference ({ci.low:.3f}, {ci.high:.3f}) t/ha")
# Student's pooled t-test assumes equal variances. How often does each test reject a TRUE null
# when the smaller group is the more variable one?
reps = 20_000
small = rng.normal(0.0, 3.0, size=(reps, 10)) # n = 10, sd 3
large = rng.normal(0.0, 1.0, size=(reps, 40)) # n = 40, sd 1, same mean
p_student = stats.ttest_ind(small, large, axis=1, equal_var=True).pvalue
p_welch = stats.ttest_ind(small, large, axis=1, equal_var=False).pvalue
print(f"\nfalse-positive rate at alpha = 0.05: Student {np.mean(p_student < 0.05):.3f}, Welch {np.mean(p_welch < 0.05):.3f}")The improved variety yielded 0.49 t/ha more on average, with a 95% confidence interval from 0.07 to 0.91 t/ha and . The interval is the main result: the improvement is probably real, but it could be anything from negligible to almost a tonne per hectare, and a seed programme would want more trials before relying on the upper end.
The second part shows why Welch's test is the better default. When the smaller group is the more variable one, Student's test rejects a true null hypothesis 25% of the time instead of 5%, because pooling the variances underestimates the noise in the small, variable group. Welch's test holds its level at 5.1%. When the variances happen to be equal, Welch's test loses almost nothing. That is why R's t.test uses Welch by default, and why statisticians recommend it routinely (Delacre, Lakens and Leys, 2017).
Paired designs
Here are the runners from the figure. Each ran once in each shoe, in random order, so the natural analysis works with each runner's difference.
import numpy as np
from scipy import stats
# Synthetic 5 km times (seconds) for 12 runners, each running once in shoe A and once in shoe B
# (in random order, on similar days). Runners differ a lot; the shoe effect is small.
rng = np.random.default_rng(8)
ability = rng.normal(1500, 120, 12) # each runner's typical time
shoe_a = np.round(ability + rng.normal(0, 8, 12))
shoe_b = np.round(ability - 10 + rng.normal(0, 8, 12))
diff = shoe_b - shoe_a
print("runner A B B - A")
for i in range(12):
print(f"{i + 1:4} {shoe_a[i]:6.0f} {shoe_b[i]:6.0f} {diff[i]:+6.0f}")
print(f"mean difference {diff.mean():.1f} s, sd of differences {diff.std(ddof=1):.1f} s")
paired = stats.ttest_rel(shoe_b, shoe_a)
unpaired = stats.ttest_ind(shoe_b, shoe_a, equal_var=False)
signed_rank = stats.wilcoxon(shoe_b, shoe_a)
print(f"\npaired t-test: t = {paired.statistic:6.2f}, p = {paired.pvalue:.4f}")
print(f"Welch test, ignoring pairs: t = {unpaired.statistic:6.2f}, p = {unpaired.pvalue:.4f}")
print(f"Wilcoxon signed-rank test: p = {signed_rank.pvalue:.4f}")
print(f"correlation between a runner's two times: {np.corrcoef(shoe_a, shoe_b)[0, 1]:.3f}")Analysed as pairs, the evidence is strong: a mean improvement of 12.2 seconds with . Analysed as if the two columns were independent groups, the same numbers give , nothing at all. The Wilcoxon signed-rank test, a rank-based version of the paired test that does not assume the differences are normal, agrees with the paired -test ().
Pairing is a choice made when designing a study, and it is one of the most powerful choices available: measuring the same patients before and after, the same plots in two seasons, or twins in a twin study. But the analysis must follow the design. Treating paired data as independent usually wastes the pairing, as here, and if the two measurements in a pair are negatively correlated it can even make results look more significant than they are. Treating independent data as paired is simply meaningless: there is no natural way to pair them.
Counts and proportions
When the outcome is a category, the data form a table of counts. A district health office ran a randomised trial: 500 families received an SMS reminder before their child's second vaccine dose was due, and 500 did not.
import numpy as np
from scipy import stats
# A randomised trial of SMS reminders: did parents bring their child back for the second dose?
# returned did not return
table = np.array([[412, 88], # SMS reminder (500 families)
[365, 135]]) # no reminder (500 families)
n1, n2 = table.sum(axis=1)
p1, p2 = table[0, 0] / n1, table[1, 0] / n2
print(f"return rates: reminder {p1:.3f}, no reminder {p2:.3f}, difference {p1 - p2:.3f}")
# Two-proportion z-test, using the pooled proportion under H0: p1 = p2.
pooled = table[:, 0].sum() / table.sum()
se0 = np.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
z = (p1 - p2) / se0
print(f"z = {z:.3f}, two-sided p = {2 * stats.norm.sf(abs(z)):.5f}")
# The chi-squared test of the same table compares observed counts with expected counts.
chi2, p, df, expected = stats.chi2_contingency(table, correction=False)
print(f"chi-squared = {chi2:.3f} (= z squared, {z**2:.3f}), df = {df}, p = {p:.5f}")
print("expected counts if reminders made no difference:\n", np.round(expected, 1))
# An effect size with its interval: the difference in proportions (unpooled standard error).
se = np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
print(f"difference {p1 - p2:.3f}, 95% CI ({p1 - p2 - 1.96 * se:.3f}, {p1 - p2 + 1.96 * se:.3f}); "
f"risk ratio {p1 / p2:.3f}")
# With small counts the chi-squared approximation is poor; Fisher's exact test is exact.
pilot = np.array([[9, 3], [3, 7]])
print("\npilot study expected counts:\n", np.round(stats.chi2_contingency(pilot)[3], 2))
print(f"Fisher's exact test p = {stats.fisher_exact(pilot).pvalue:.4f}")With reminders, 82.4% of children came back, against 73.0% without, a difference of 9.4 percentage points. The two-proportion -test and the chi-squared test are the same test in different clothes (, ), and both compare the observed counts with the counts expected if reminders made no difference (388.5 returns in each arm).
The report a health office can act on is the effect size: reminders raised the return rate by 9.4 points (95% CI 4.3 to 14.5), a risk ratio of 1.13. Translated into children, that is roughly 9 more returns for every 100 families sent a message, which, set against the cost of the messages, is what decides whether to adopt them. For rare outcomes, epidemiologists also report odds ratios, which logistic regression estimates (in Generalised linear models).
The chi-squared test relies on a large-sample approximation. The usual rule asks for expected counts of at least 5 in every cell; the pilot study's smallest expected count is 4.55, so we use Fisher's exact test instead, which computes the p-value exactly from all tables with the same margins (). The chi-squared test also extends to bigger tables, such as preferred transport mode (bus, minibus, motorcycle, walking) by city, with degrees of freedom. For paired yes/no data, such as the same people asked before and after a campaign, the right tool is McNemar's test, which uses only the people whose answer changed.
Without the normal assumption: ranks and permutations
-tests lean on the central limit theorem, and with small, skewed samples that support can be weak, as Sampling distributions and the CLT showed. Two families of tests avoid the normal assumption.
A permutation test builds the null distribution from the data themselves. If the ward made no difference to length of stay, the labels "ward A" and "ward B" are arbitrary: any shuffling of the 33 stays into groups of 15 and 18 was as likely as the one observed. So we shuffle thousands of times, recompute the difference in means each time, and see where the real difference falls. The idea is Ronald Fisher's, from the 1930s, but it became practical only with computers. Here it is, implemented from scratch, alongside Welch's test and the Mann–Whitney test on the same data, followed by a simulation of what happens when an analyst runs several tests and reports the best.
import numpy as np
from scipy import stats
# Synthetic lengths of hospital stay (days) on two wards: skewed, with a few long stays.
rng = np.random.default_rng(34)
ward_a = np.round(rng.lognormal(1.3, 0.6, 15), 1)
ward_b = np.round(rng.lognormal(1.75, 0.6, 18), 1)
print("ward A:", np.sort(ward_a))
print("ward B:", np.sort(ward_b))
observed = ward_b.mean() - ward_a.mean()
print(f"difference in means (B - A): {observed:.2f} days")
def permutation_test(a, b, n_perm, rng):
"""Two-sided permutation test for a difference in means."""
pooled = np.concatenate([a, b])
obs = abs(b.mean() - a.mean())
count = 0
for _ in range(n_perm):
shuffled = rng.permutation(pooled) # relabel at random, as H0 allows
diff = shuffled[a.size:].mean() - shuffled[:a.size].mean()
count += abs(diff) >= obs
return (count + 1) / (n_perm + 1) # count the observed labelling too
print(f"permutation test (20,000 relabellings): p = {permutation_test(ward_a, ward_b, 20_000, rng):.4f}")
print(f"Welch t-test: p = {stats.ttest_ind(ward_b, ward_a, equal_var=False).pvalue:.4f}")
print(f"Mann-Whitney U test: p = {stats.mannwhitneyu(ward_b, ward_a).pvalue:.4f}")
# The forking path: run four tests on data with NO real difference, and keep the smallest p.
reps = 4_000
a = rng.lognormal(1.5, 0.8, size=(reps, 15))
b = rng.lognormal(1.5, 0.8, size=(reps, 18))
ps = np.vstack([
stats.ttest_ind(a, b, axis=1, equal_var=False).pvalue,
stats.ttest_ind(a, b, axis=1, equal_var=True).pvalue,
stats.mannwhitneyu(a, b, axis=1).pvalue,
stats.ttest_ind(np.log(a), np.log(b), axis=1).pvalue,
])
print(f"\nfalse-positive rate of each test alone: {np.round(np.mean(ps < 0.05, axis=1), 3)}")
print(f"false-positive rate of 'whichever test gives the smallest p': {np.mean(ps.min(axis=0) < 0.05):.3f}")The three tests disagree, and the disagreement is instructive. Ward A has one patient who stayed 17.2 days. That single long stay raises ward A's mean and inflates its variance, so Welch's test gives . The permutation test, which compares the same means but builds its null distribution by reshuffling rather than from the distribution, gives 0.040. The Mann–Whitney test works with ranks, so it does not care whether the long stay was 17.2 days or 12: it sees that ward B's stays mostly rank above ward A's, and gives 0.005. They answer different questions. Welch's and the permutation test ask about the mean length of stay, which is what drives bed-days and costs; Mann–Whitney asks whether a typical patient on ward B stays longer than a typical patient on ward A.
The second part shows why the choice must come first. On data with no real difference, each test alone gives false positives at about its nominal rate (3.8% to 5.0%). An analyst who runs all four and reports the smallest p-value gets a false-positive rate of 7.3%. The tests are highly correlated, so the inflation is modest here; with more tests, more outcomes and more subgroups the forking paths multiply quickly.
Choose the test by the design
The design and the question, not the p-value, should pick the test. The guide below covers the cases in this lesson; when the outcome is a measurement and the samples are small and strongly skewed, the bracketed alternatives are worth planning for in advance.
Three further points make the guide work in practice.
- Decide what you want to compare. Means (for totals and costs), medians or typical values (for skewed outcomes such as income or waiting time), or proportions. This, more than distributional assumptions, decides between Welch and Mann–Whitney.
- Write it down before looking. The test, the outcome, the comparison and any transformations go in the analysis plan. If the data later turn out to violate an assumption badly, say so and report both analyses, rather than quietly switching.
- More than two groups is a different problem. Comparing four fertilisers by running six pairwise -tests inflates false positives; the methods for that, analysis of variance and its follow-up comparisons, are in Experimental design and ANOVA. And every test in this lesson is a special case of a regression model, which is how they are usually extended to adjust for other variables (Linear regression onwards).
Comparing groups in professional practice
Clinical trials compare treatments under a protocol written in advance, report the primary comparison with its confidence interval, and follow reporting guidelines such as CONSORT. Online experiments in technology companies compare conversion rates or revenue between randomised groups of users, at enormous scale; the Data Science elective's experiments in products covers their practical pitfalls. Agricultural field trials, where many of these methods were born, compare varieties on plots arranged in blocks so that plots in similar soil are compared with each other, the field version of pairing.
Effect sizes for comparisons. Report the difference in means (in real units) with its interval; for proportions, the risk difference and the risk ratio (and the number of people who need to be treated for one to benefit, , about 11 families per extra return in the SMS trial). Standardised effect sizes such as Cohen's help when combining studies measured on different scales.
Software. SciPy provides ttest_ind (pass equal_var=False for Welch), ttest_rel, wilcoxon, mannwhitneyu, chi2_contingency, fisher_exact and a general permutation_test. In R, shown for comparison and not run:
# R, not run here
t.test(yield ~ variety, data = plots) # Welch by default
t.test(shoe_b, shoe_a, paired = TRUE)
wilcox.test(stay ~ ward, data = stays) # Mann-Whitney
chisq.test(matrix(c(412, 365, 88, 135), 2), correct = FALSE)
fisher.test(matrix(c(9, 3, 3, 7), 2))History. Karl Pearson introduced the chi-squared test in 1900. Student's (1908) was extended to unequal variances by Bernard Welch in 1947, the same year Henry Mann and Donald Whitney published their rank test (Frank Wilcoxon had proposed an equivalent test in 1945). Fisher described the logic of permutation tests in the 1930s, when the reshuffling had to be done by hand for tiny samples; today a laptop does tens of thousands of reshuffles in about a second.
Common misconceptions
Misconception
"Check normality and equal variances first, then choose the test." Two-stage procedures distort error rates, and preliminary tests have little power in small samples. Choose by design and question in advance; use Welch's test as the default for independent means.
Misconception
"The Mann–Whitney test compares medians." It tests whether values in one group tend to exceed those in the other, . Only if the two distributions have the same shape and differ by a shift is that the same as comparing medians. Two groups with equal medians but different shapes can give a significant Mann–Whitney test.
Misconception
"If two confidence intervals overlap, the difference is not significant." Two 95% intervals can overlap noticeably while the difference between the groups is significant at 5%, because the standard error of a difference is smaller than the sum of the two standard errors. Compute the interval for the difference itself.
Misconception
"Treatment worked in women () but not in men (), so it works differently by sex." Suppose the effect is 2.0 (standard error 0.8) in one group and 0.8 (standard error 0.8) in the other. The difference between the two effects is 1.2 with standard error 1.13, giving . The difference between "significant" and "not significant" is not itself significant (Gelman and Stern, 2006); to claim the groups differ, test the difference.
Misconception
"Analysing paired data as independent just loses a little power." It uses the wrong standard error. With positively correlated pairs it usually throws away most of the information (in the shoe example, instead of ); with negatively correlated pairs it can overstate the evidence.
Exercises
1. Comparing groups
Warm-upHumanA clinic in Kumasi measures the systolic blood pressure of 30 patients, then again after each has followed a three-month exercise programme. Which analysis matches the design?
2. Comparing groups
CoreHumanA survey of 200 adults in Tanzania records where they live and whether they use mobile money.
| uses mobile money | does not | total | |
|---|---|---|---|
| urban | 102 | 18 | 120 |
| rural | 48 | 32 | 80 |
| total | 150 | 50 | 200 |
For the chi-squared test of independence, what is the expected number of urban adults who use mobile money?
3. Comparing groups
CoreHumanAn exact permutation test compares two groups of 5 measurements each by considering every way of splitting the 10 pooled values into a group of 5 labelled A and a group of 5 labelled B. How many splits are there?
4. Comparing groups
CoreHumanA Mann–Whitney test comparing waiting times at two clinics gives . Which conclusion does the test itself support?
5. Comparing groups
CoreHumanAn analyst comparing two groups runs Welch's test (), Student's test (), a Mann–Whitney test () and a -test on logarithms (). She reports only the Mann–Whitney result as "significant at the 5% level". What is the main problem?
6. Comparing groups
CoreComputationalWrite permutation_p_value(a, b, n_perm, seed), a two-sided permutation test for a difference in means:
- Pool
aandb. Compute the observed statisticabs(mean(b) - mean(a)). - With
rng = np.random.default_rng(seed), repeatn_permtimes: shuffle the pooled values (rng.permutation), call the firstlen(a)of them group A and the rest group B, and compute the same statistic. - Count the reshuffles whose statistic is at least the observed one (compare with
>= observed - 1e-9, which protects against rounding), and return(count + 1) / (n_perm + 1), which counts the observed labelling as one of the possibilities.
7. Comparing groups
StretchComputationalWrite welch_test(a, b) returning a tuple of three floats (t, df, p): Welch's statistic for mean(b) - mean(a), the Welch–Satterthwaite degrees of freedom, and the two-sided p-value. Use the sample variances (with ddof=1) and scipy.stats.t.sf for the tail probability, but not ttest_ind itself. The tests compare your results with SciPy's.
Further reading and resources
- The generalization of "Student's" problem when several different population variances are involved · B. L. Welch · 1947The origin of Welch's t-test.
- Why psychologists should by default use Welch's t-test instead of Student's t-test · Marie Delacre, Daniël Lakens and Christophe Leys · 2017A clear simulation study of when Student's test goes wrong, with practical advice that applies well beyond psychology.
- On a test of whether one of two random variables is stochastically larger than the other · H. B. Mann and D. R. Whitney · 1947
- On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling · Karl Pearson · 1900The paper that introduced the chi-squared test.
- The difference between "significant" and "not significant" is not itself statistically significant · Andrew Gelman and Hal Stern · 2006
- SciPy: scipy.stats.permutation_test
- CONSORT and SPIRIT reporting guidelinesHow randomised trials should be reported, including the comparison of groups.