Skip to content
LumenaryAcademy
Foundations 70 min

Hypothesis testing

Could it be chance? The logic of significance tests, what a p-value is and is not, type I and II errors and power computed by simulation, effect sizes, and why "not significant" never means "no effect".

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Could it be chance?

A textile factory in Bursa buys zips from a supplier whose contract promises that no more than 2% are defective. An inspector tests 200 zips from a new delivery and finds 9 defective. At 2%, she would expect 4. Should the factory reject the delivery and complain?

The supplier's reply is easy to predict: "Defects come in random numbers. Sometimes there are a few more, sometimes a few less. You were unlucky." The supplier might be right. The question the factory needs answered is: if the supplier's claim were true, how surprising would 9 defects in 200 be? If such a result would happen often by chance alone, the inspection is no evidence against the supplier. If it would almost never happen, then either something very unusual occurred or the claim is false.

That question, "could it be chance?", is the heart of a hypothesis test, the most widely used and most widely misunderstood tool in statistics. Tests decide which drugs are approved, which features ship in an app, whether a fertiliser works, and whether a particle has been discovered. Used well, they guard against the human habit of seeing patterns in noise. Used badly, they have filled journals with findings that do not replicate. This lesson teaches the logic, the arithmetic and the limits, honestly.

IntuitionH

Proof by contradiction, with probability

A test argues like a proof by contradiction, softened by probability. Start by assuming the boring explanation, the null hypothesis: the supplier's defect rate really is 2%. Work out what the data would look like if that were true: the counts of defects you would typically see in 200 zips. Then compare the actual data. If they sit comfortably among the typical results, there is no contradiction. If they lie far out in the tail, the assumption is under strain.

The strength of that strain is the p-value: the probability, computed assuming the null hypothesis is true, of getting a result at least as extreme as the one observed. A small p-value means "data like these would be rare if the null hypothesis were true". It does not mean "the null hypothesis is probably false", although the two are endlessly confused. We will come back to that.

Analogy, not a formal construction

A criminal court presumes innocence and convicts only when the evidence would be very hard to explain if the accused were innocent. An acquittal does not declare the accused innocent; it says the evidence was not strong enough. In the same way, a test that fails to reject the null hypothesis has not shown the null hypothesis to be true. The analogy has limits (courts do not compute probabilities), but its asymmetry is exactly right.

DefinitionsH

Definitions

Definition— null and alternative hypotheses

The null hypothesis H0H_0 is a precise statement about the population, usually "no effect" or "the claimed value": p=0.02p = 0.02, or μ=10\mu = 10. The alternative hypothesis H1H_1 is what we would conclude instead: p>0.02p > 0.02 (one-sided) or μ≠10\mu \ne 10 (two-sided).

Definition— test statistic and null distribution

A test statistic is a number computed from the data that measures departure from H0H_0, such as a count of defects or t=(xˉ−μ0)/(s/n)t = (\bar x - \mu_0)/(s/\sqrt n). Its distribution when H0H_0 is true is the null distribution.

Definition— p-value

The p-value is the probability, under H0H_0 and the model's other assumptions, of a test statistic at least as extreme as the one observed, where "extreme" means "in the direction of H1H_1".

Definition— significance level, errors and power

A significance level α\alpha (often 0.05) is a threshold fixed in advance: reject H0H_0 if p≤αp \le \alpha. Rejecting a true H0H_0 is a type I error (a false positive); failing to reject a false H0H_0 is a type II error (a false negative). The test is designed so that P(type I error)≤αP(\text{type I error}) \le \alpha. The power of the test against a particular alternative is P(reject H0)P(\text{reject } H_0) when that alternative is true, which equals 1−β1 - \beta, where β\beta is the type II error rate.

Definition— effect size

An effect size measures how large a departure from H0H_0 is, in units that mean something: a difference in means (in hours, or kilograms), a ratio of risks, or a standardised difference such as Cohen's dd, the difference in means divided by the standard deviation.

ComputationC

A first test, computed two ways

For the zips, the null hypothesis is p=0.02p = 0.02 and the alternative is p>0.02p > 0.02: the factory cares only whether the supplier is worse than promised. The test statistic is the number of defects XX, whose null distribution is Binomial(200, 0.02). The p-value is P(X≥9)P(X \ge 9) under that distribution. We compute it exactly, and also by brute force, simulating a world in which the supplier is honest.

python
import numpy as np
from scipy import stats

n, p0, observed = 200, 0.02, 9      # the supplier's claim: 2% defective; the inspector found 9 in 200

# The p-value: if the claim were true, how often would an inspection find 9 or more defects?
exact = stats.binom.sf(observed - 1, n, p0)          # P(X >= 9) = P(X > 8)
print(f"exact p-value  P(X >= {observed} | p = {p0}) = {exact:.4f}")

# The same by brute force: simulate 100,000 inspections in a world where the claim is true.
rng = np.random.default_rng(5)
null_world = rng.binomial(n, p0, size=100_000)
print(f"simulated p-value: {np.mean(null_world >= observed):.4f}")
print("SciPy's binomtest agrees:", round(stats.binomtest(observed, n, p0, alternative="greater").pvalue, 4))

print("\nnull distribution of the defect count (# = 1% of inspections):")
for k in range(13):
    share = np.mean(null_world == k)
    mark = "   <- as or more extreme than observed" if k >= observed else ""
    print(f"{k:2} {share:6.3f} " + "#" * round(100 * share) + mark)
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
0.050.100.150.20012345678910111213149 or more: total 0.020(the p-value)number of defective parts in 200, if the true rate is 2%
The null distribution for the zip inspection: how many defects 200 zips would contain if the true rate were 2%. The highlighted bars, 9 or more defects, add up to 0.020. That total is the p-value.

The p-value is 0.020: if the supplier's claim were true, only about 2 inspections in 100 would find 9 or more defects. The simulation, which knows nothing about binomial formulas, agrees to within its own sampling error. Since 0.020≤0.050.020 \le 0.05, at the conventional level the factory would reject the claim.

What has not been shown matters as much. We have not shown that the defect rate is 4.5% (the sample's rate): a 95% Wilson interval for it runs from about 2.4% to 8.3%. We have not computed the probability that the supplier is cheating. And the evidence is moderate, not overwhelming: a p-value of 0.02 is the kind of result that appears now and then even when nothing is wrong.

Worked examplesH

A test for a mean, and its link to intervals

Most tests follow the same pattern: (estimate − hypothesised value) divided by a standard error, compared with a known distribution.

Example— Do the solar lanterns last 10 hours?

A manufacturer advertises that its solar lanterns give 10 hours of light on a full charge. A consumer group in Dhaka tests 20 lanterns and records a mean of xˉ=9.6\bar x = 9.6 hours with standard deviation s=0.8s = 0.8. Test H0 ⁣:μ=10H_0\colon \mu = 10 against H1 ⁣:μ≠10H_1\colon \mu \ne 10.

The standard error is 0.8/20=0.1790.8/\sqrt{20} = 0.179, so t=9.6−100.179=−2.24,t = \frac{9.6 - 10}{0.179} = -2.24, with n−1=19n - 1 = 19 degrees of freedom. The two-sided p-value is P(∣T19∣≥2.24)=0.038P(|T_{19}| \ge 2.24) = 0.038. The 95% confidence interval is 9.6±2.093×0.179=(9.23,9.97)9.6 \pm 2.093 \times 0.179 = (9.23, 9.97) hours, and the effect size is d=(9.6−10)/0.8=−0.5d = (9.6 - 10)/0.8 = -0.5, half a standard deviation.

Notice that the confidence interval excludes 10 and the two-sided p-value is below 0.05. That is no coincidence. A 95% confidence interval contains exactly the hypothesised values that a two-sided test at level 0.05 would not reject. This duality is why many statisticians prefer to report the interval: it shows the test's verdict for every hypothesised value at once, and the size of the effect too.

One-sided or two-sided? The one-sided p-value for H1 ⁣:μ<10H_1\colon \mu < 10 is half the two-sided one, 0.019. Halving the p-value is tempting, which is exactly why the choice must be made before seeing the data and justified by the question. A one-sided test is honest only if an effect in the other direction would lead to the same decision as no effect at all. A consumer group that would also report lanterns lasting much longer than advertised should use a two-sided test. Switching to one-sided after seeing which way the data fell doubles the real type I error rate.

ExplanationH

Errors, significance and power

Every test decision can go wrong in two ways.

H0H_0 trueH0H_0 false
Reject H0H_0type I error (false positive), probability α\alphacorrect, probability = power
Do not reject H0H_0correcttype II error (false negative), probability β\beta

The significance level controls the first row's left cell: the test is built so that, if H0H_0 is true, it rejects with probability α\alpha. The simulation below checks this, and shows how easily it is broken. It runs 10,000 experiments in which the null hypothesis is exactly true, then repeats them with a researcher who "peeks" at the data after every 10 observations and stops as soon as p<0.05p < 0.05.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(21)
reps, n = 10_000, 100

# 10,000 experiments in which the null hypothesis is TRUE: the mean really is 0.
data = rng.normal(0.0, 1.0, size=(reps, n))
p = stats.ttest_1samp(data, 0.0, axis=1).pvalue
print(f"share of p-values below 0.05: {np.mean(p < 0.05):.4f}   (the type I error rate)")
print("p-values under a true null, in ten bins of width 0.1:")
counts, _ = np.histogram(p, bins=10, range=(0, 1))
print(counts)

# Peeking: test after every 10 observations and stop as soon as p < 0.05.
looks = range(10, n + 1, 10)
stopped_early = np.zeros(reps, dtype=bool)
for m in looks:
    p_m = stats.ttest_1samp(data[:, :m], 0.0, axis=1).pvalue
    stopped_early |= p_m < 0.05
print(f"\nwith 10 peeks, share of experiments that ever reach p < 0.05: {np.mean(stopped_early):.4f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With a single, pre-planned test, 5.2% of the true-null experiments give p<0.05p < 0.05, as promised (within simulation error), and the p-values are spread evenly between 0 and 1: under a true null, a p-value is uniformly distributed. That is what "type I error rate α\alpha" means. With ten peeks, 20% of the experiments reach significance at some point, four times the nominal rate, although every single test was computed correctly. This is one of the forking paths by which honest researchers produce false findings: choosing when to stop, which outcome to report, which subgroup to analyse or which outliers to drop, after seeing the data. Sequential designs that allow planned interim looks exist, with adjusted thresholds, and are standard in clinical trials; unplanned peeking is not one of them.

reject H₀ to the right →if H₀ is trueif the effect is realαβpower-2041.645value of the test statistic
The two kinds of error for a one-sided test at α = 0.05. If the null hypothesis is true, the statistic follows the solid curve, and it lands beyond the critical value 1.645 with probability α. If there is a real effect, the statistic follows the dashed curve; the part beyond the critical value is the power (here 0.80) and the rest is β, the chance of missing the effect.

Power is the other side. A test with low power usually misses real effects, so a study should be designed to have enough power for the smallest effect worth detecting. Power grows with the true effect size, with the sample size, and with α\alpha, and falls as the data get noisier. It can be computed by formula, or simply by simulating the study many times in a world where the effect is real, which works for any design.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(13)

def power_by_simulation(effect, n, alpha=0.05, reps=10_000):
    """Share of two-sided one-sample t-tests that reject when the true mean is `effect` SDs from 0."""
    data = rng.normal(effect, 1.0, size=(reps, n))
    return np.mean(stats.ttest_1samp(data, 0.0, axis=1).pvalue < alpha)

def power_by_formula(effect, n, alpha=0.05):
    """The same power from the noncentral t distribution."""
    df = n - 1
    crit = stats.t.ppf(1 - alpha / 2, df)
    nc = effect * np.sqrt(n)
    return stats.nct.sf(crit, df, nc) + stats.nct.cdf(-crit, df, nc)

print("power to detect a mean difference of 0.5 standard deviations (alpha = 0.05, two-sided)")
print("   n   simulated   formula")
for n in [10, 20, 34, 50, 100]:
    print(f"{n:4} {power_by_simulation(0.5, n):11.3f} {power_by_formula(0.5, n):9.3f}")

print("\npower with n = 34 for different true effects")
for d in [0.0, 0.2, 0.35, 0.5, 0.8]:
    print(f"effect {d:4}: {power_by_formula(d, 34):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
ExplanationH

What p-values are not

By the 2010s, misuse of p-values had become serious enough that the American Statistical Association did something unprecedented: in 2016 it issued a formal statement on them (Wasserstein and Lazar, 2016). Its six principles are worth reading in the original. In summary, and in our words: a p-value measures how incompatible the data are with a particular statistical model; it is not the probability that the hypothesis is true, nor the probability that the results arose "by chance"; scientific and policy decisions should not rest only on whether a p-value crosses a threshold; honest inference needs full reporting of every analysis performed; a p-value does not measure the size or importance of an effect; and on its own a p-value is weak evidence for or against a hypothesis.

The commonest misreadings, with their corrections:

  • "p=0.02p = 0.02 means a 2% chance that H0H_0 is true." The p-value is computed assuming H0H_0; it is P(data this extreme∣H0)P(\text{data this extreme} \mid H_0), not P(H0∣data)P(H_0 \mid \text{data}). Turning one into the other needs Bayes' theorem and a prior, as in Bayesian inference. When most hypotheses tested in a field are false leads, a large share of "significant" results can be false positives even at p<0.05p < 0.05.
  • "A small p-value means a large or important effect." With enough data, a trivially small effect gives a tiny p-value. A drug that lowers blood pressure by 0.3 mmHg in a trial of 200,000 people would be highly significant and clinically useless.
  • "p=0.06p = 0.06 and p=0.04p = 0.04 tell different stories." They are nearly the same evidence. The 0.05 line is a convention, not a law of nature.

Statisticians agree on these points and disagree, sometimes sharply, about the remedy. One group proposed lowering the default threshold for claims of new discoveries to 0.005 (Benjamin et al., 2018). Others argued for abandoning the language of "statistical significance" altogether in favour of reporting estimates and intervals and interpreting them in context (Amrhein, Greenland and McShane, 2019). Many journals now require effect sizes and confidence intervals alongside any p-value. In fields that test sharply defined predictions, such as particle physics, very strict thresholds are the norm (the "five sigma" standard corresponds to a one-sided p-value of about 3×10−73 \times 10^{-7}).

The stakes became visible in the replication crisis. In 2015 the Open Science Collaboration repeated 100 published psychology experiments as closely as possible: 97% of the originals had reported significant results, but only about 36% of the replications did, and the replication effects were on average about half as large (Open Science Collaboration, 2015). Similar projects in economics, cancer biology and other fields found varied but sobering rates. The causes were not mainly fraud. They were small samples, flexible analyses and a publication system that rewarded p<0.05p < 0.05, exactly the mechanisms this lesson simulates. The response has been constructive: preregistration, registered reports, larger collaborative studies and open data have spread rapidly, and the conversation about evidence is healthier for it.

ExplanationH

Not significant is not no effect

"The difference was not statistically significant" is often reported as "there was no difference". The simulation below studies a real effect of 0.3 standard deviations with samples of 20, and looks at what such studies report.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(99)
true_effect, n, reps = 0.3, 20, 10_000     # a real but modest effect, studied with small samples

data = rng.normal(true_effect, 1.0, size=(reps, n))
result = stats.ttest_1samp(data, 0.0, axis=1)
estimates = data.mean(axis=1)
significant = result.pvalue < 0.05

print(f"true effect {true_effect}; share of studies with p < 0.05 (the power): {significant.mean():.3f}")
print(f"average estimate, all studies:          {estimates.mean():.3f}")
print(f"average estimate, significant ones only: {estimates[significant].mean():.3f}")

# One of the non-significant studies, reported properly with its confidence interval.
i = np.flatnonzero(~significant)[0]
ci = stats.ttest_1samp(data[i], 0.0).confidence_interval(0.95)
print(f"\nstudy {i}: estimate {estimates[i]:.2f}, p = {result.pvalue[i]:.2f}, "
      f"95% CI ({ci.low:.2f}, {ci.high:.2f})")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The effect is real in every one of the 10,000 studies, yet only 24% reach p<0.05p < 0.05. A study like number 2, with estimate 0.35 and p=0.10p = 0.10, has not shown that the effect is absent. Its confidence interval, from −0.07-0.07 to 0.770.77, includes zero, but it also includes effects more than twice the true size. The honest summary is "the data are consistent with no effect and with a large one; this study was too small to tell". Absence of evidence is not evidence of absence.

The middle lines show a subtler danger. Averaged over all studies, the estimate is unbiased (0.299). Averaged over the significant studies only, it is 0.570, nearly double the truth, because with n=20n = 20 only the samples that happened to overestimate the effect clear the threshold. If journals publish mainly significant results, the published literature systematically exaggerates effects. Andrew Gelman and John Carlin call this a type M (magnitude) error (2014), and it explains much of the shrinkage seen in replications.

To show that an effect is absent, or at least too small to matter, you need a different design: decide in advance the smallest effect that would matter, then check whether the confidence interval lies entirely inside that margin. This is an equivalence test (Lakens, 2017), and it is how regulators judge whether a generic medicine is equivalent to the original.

ExplanationH

Testing in professional practice

Plan before you look. Good practice fixes the hypotheses, the primary outcome, the test, the sample size (from a power calculation) and the stopping rule before any data are seen, and records them publicly. Clinical trials must be registered in advance, in registries such as ClinicalTrials.gov and the WHO's International Clinical Trials Registry Platform. In other sciences, preregistration and registered reports (journals accept a study on its design, before the results are known) play the same role.

Report the effect, its interval, and everything you tried. A p-value without an estimate hides the size of the effect; an analysis reported without the others that were run hides the forking paths. Testing many hypotheses at once needs its own corrections, the subject of Multiple testing and reproducibility.

Tests in industry. Technology companies run thousands of randomised experiments (A/B tests) a year to decide which version of a product to ship; the methods, and the same peeking problem, are covered in the Data Science elective's experiments in products. Manufacturers use tests in acceptance sampling, as in the zip example, and in control charts.

Software. In Python, scipy.stats has binomtest, ttest_1samp and dozens of other tests, each returning a statistic and a p-value (and, for many, a confidence interval). In R, shown here for comparison and not run:

r
# R, not run here
binom.test(9, 200, p = 0.02, alternative = "greater")
t.test(hours, mu = 10)                                  # two-sided one-sample t-test
power.t.test(delta = 0.5, sd = 1, power = 0.8, type = "one.sample")

History. In the 1920s Ronald Fisher, at the Rothamsted agricultural research station in England, made the p-value popular as a continuous measure of evidence against a null hypothesis. His famous illustration was a colleague, Muriel Bristol, who said she could taste whether milk had been poured into the cup before or after the tea; Fisher designed a test with eight cups, four of each kind, in which guessing would identify all four milk-first cups with probability only 1/701/70. Jerzy Neyman and Egon Pearson then framed testing as a decision rule with two kinds of error and a power to be maximised (1933). Modern practice is an uneasy hybrid of the two views, which is one root of the confusion this lesson tries to clear up.

Common misconceptionsH

Common misconceptions

Misconception

"The p-value is the probability that the null hypothesis is true." It is the probability of data at least this extreme if the null hypothesis is true. These can differ enormously, depending on how plausible the hypothesis was to begin with.

Misconception

"Not significant means no effect." It means the data could not rule out zero, which is also what happens when a real effect is studied with too little data. Report the interval, and use an equivalence test to show an effect is negligible.

Misconception

"Significant means important." Statistical significance measures how clearly the data depart from the null model, which depends heavily on sample size. Importance depends on the size of the effect and the context: a tiny effect can be significant, and a large one can fail to be.

Misconception

"If p=0.01p = 0.01, a replication has a 99% chance of being significant." The probability of a significant replication depends on the true effect and the replication's power, and is often far lower; the significance filter also means the original estimate is likely to be too large.

Misconception

"It is fine to look at the data, then pick the test (or the side, or the stopping point) that gives the smallest p-value." Each choice made after seeing the data inflates the type I error rate, as the peeking simulation showed (20% instead of 5%). Choose by the design and the question, in advance.

Exercises

Exercises

1. Hypothesis testing

Warm-upHuman

A trial compares a new cough syrup with a placebo and reports a difference in mean recovery time with p=0.03p = 0.03. Which statement is correct?

Choose one

2. Hypothesis testing

Warm-upHuman

A genetics lab tests 200 genes, one test each at α=0.05\alpha = 0.05. Suppose that in truth none of the genes is linked to the disease. How many genes would you expect to be declared significant anyway?

3. Hypothesis testing

CoreHuman

In the zip example, the null hypothesis is that the defect rate is 2%, and the inspector tests 200 zips with a one-sided test at α=0.05\alpha = 0.05. What is the smallest number of defects that would lead her to reject the null hypothesis?

Useful values for X∼Binomial(200,0.02)X \sim \text{Binomial}(200, 0.02): P(X≥6)=0.213P(X \ge 6) = 0.213, P(X≥7)=0.109P(X \ge 7) = 0.109, P(X≥8)=0.049P(X \ge 8) = 0.049, P(X≥9)=0.020P(X \ge 9) = 0.020.

4. Hypothesis testing

CoreHuman

A researcher planned a two-sided test of whether a new teaching app changes exam scores. The result is p=0.08p = 0.08, with scores higher in the app group. She then reports a one-sided test ("the app improves scores"), p=0.04p = 0.04, significant at 0.05. What is wrong?

Choose one

5. Hypothesis testing

CoreHuman

A small trial of a salt-reduction programme in 60 adults finds that systolic blood pressure fell by 2.8 mmHg more in the programme group than in the control group, with 95% confidence interval (−1.2,6.8)(-1.2, 6.8) mmHg and p=0.16p = 0.16. Which conclusion is best supported?

Choose one

6. Hypothesis testing

CoreHuman

Using the power table from the lesson's simulation (two-sided one-sample tt-test, α=0.05\alpha = 0.05, true effect 0.5 standard deviations), what is the smallest sample size in the table that gives at least 80% power?

nn 10 20 34 50 100
power 0.293 0.565 0.808 0.934 0.999

7. Hypothesis testing

CoreComputational

Write p_value_at_least(k, n, p0), the one-sided p-value P(X≥k)P(X \ge k) for X∼Binomial(n,p0)X \sim \text{Binomial}(n, p_0), using only math.comb (no SciPy). The starter makes a common mistake: it returns P(X=k)P(X = k), the probability of the observed count alone, instead of the whole tail of results at least as extreme.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Hypothesis testing

StretchComputational

Write simulated_power(effect, n, reps, seed, alpha=0.05). With rng = np.random.default_rng(seed), simulate reps studies, each a sample of n values from a normal distribution with mean effect and standard deviation 1. For each study run a two-sided one-sample tt-test of mean 0 (scipy.stats.ttest_1samp accepts axis=1 to test every row at once). Return, as a float, the share of studies with p-value below alpha.

With effect=0 you are measuring the type I error rate, which should be close to alpha.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources