Could it be chance?
A textile factory in Bursa buys zips from a supplier whose contract promises that no more than 2% are defective. An inspector tests 200 zips from a new delivery and finds 9 defective. At 2%, she would expect 4. Should the factory reject the delivery and complain?
The supplier's reply is easy to predict: "Defects come in random numbers. Sometimes there are a few more, sometimes a few less. You were unlucky." The supplier might be right. The question the factory needs answered is: if the supplier's claim were true, how surprising would 9 defects in 200 be? If such a result would happen often by chance alone, the inspection is no evidence against the supplier. If it would almost never happen, then either something very unusual occurred or the claim is false.
That question, "could it be chance?", is the heart of a hypothesis test, the most widely used and most widely misunderstood tool in statistics. Tests decide which drugs are approved, which features ship in an app, whether a fertiliser works, and whether a particle has been discovered. Used well, they guard against the human habit of seeing patterns in noise. Used badly, they have filled journals with findings that do not replicate. This lesson teaches the logic, the arithmetic and the limits, honestly.
Proof by contradiction, with probability
A test argues like a proof by contradiction, softened by probability. Start by assuming the boring explanation, the null hypothesis: the supplier's defect rate really is 2%. Work out what the data would look like if that were true: the counts of defects you would typically see in 200 zips. Then compare the actual data. If they sit comfortably among the typical results, there is no contradiction. If they lie far out in the tail, the assumption is under strain.
The strength of that strain is the p-value: the probability, computed assuming the null hypothesis is true, of getting a result at least as extreme as the one observed. A small p-value means "data like these would be rare if the null hypothesis were true". It does not mean "the null hypothesis is probably false", although the two are endlessly confused. We will come back to that.
Analogy, not a formal construction
A criminal court presumes innocence and convicts only when the evidence would be very hard to explain if the accused were innocent. An acquittal does not declare the accused innocent; it says the evidence was not strong enough. In the same way, a test that fails to reject the null hypothesis has not shown the null hypothesis to be true. The analogy has limits (courts do not compute probabilities), but its asymmetry is exactly right.
Definitions
Definition— null and alternative hypotheses
The null hypothesis is a precise statement about the population, usually "no effect" or "the claimed value": , or . The alternative hypothesis is what we would conclude instead: (one-sided) or (two-sided).
Definition— test statistic and null distribution
A test statistic is a number computed from the data that measures departure from , such as a count of defects or . Its distribution when is true is the null distribution.
Definition— p-value
The p-value is the probability, under and the model's other assumptions, of a test statistic at least as extreme as the one observed, where "extreme" means "in the direction of ".
Definition— significance level, errors and power
A significance level (often 0.05) is a threshold fixed in advance: reject if . Rejecting a true is a type I error (a false positive); failing to reject a false is a type II error (a false negative). The test is designed so that . The power of the test against a particular alternative is when that alternative is true, which equals , where is the type II error rate.
Definition— effect size
An effect size measures how large a departure from is, in units that mean something: a difference in means (in hours, or kilograms), a ratio of risks, or a standardised difference such as Cohen's , the difference in means divided by the standard deviation.
A first test, computed two ways
For the zips, the null hypothesis is and the alternative is : the factory cares only whether the supplier is worse than promised. The test statistic is the number of defects , whose null distribution is Binomial(200, 0.02). The p-value is under that distribution. We compute it exactly, and also by brute force, simulating a world in which the supplier is honest.
import numpy as np
from scipy import stats
n, p0, observed = 200, 0.02, 9 # the supplier's claim: 2% defective; the inspector found 9 in 200
# The p-value: if the claim were true, how often would an inspection find 9 or more defects?
exact = stats.binom.sf(observed - 1, n, p0) # P(X >= 9) = P(X > 8)
print(f"exact p-value P(X >= {observed} | p = {p0}) = {exact:.4f}")
# The same by brute force: simulate 100,000 inspections in a world where the claim is true.
rng = np.random.default_rng(5)
null_world = rng.binomial(n, p0, size=100_000)
print(f"simulated p-value: {np.mean(null_world >= observed):.4f}")
print("SciPy's binomtest agrees:", round(stats.binomtest(observed, n, p0, alternative="greater").pvalue, 4))
print("\nnull distribution of the defect count (# = 1% of inspections):")
for k in range(13):
share = np.mean(null_world == k)
mark = " <- as or more extreme than observed" if k >= observed else ""
print(f"{k:2} {share:6.3f} " + "#" * round(100 * share) + mark)The p-value is 0.020: if the supplier's claim were true, only about 2 inspections in 100 would find 9 or more defects. The simulation, which knows nothing about binomial formulas, agrees to within its own sampling error. Since , at the conventional level the factory would reject the claim.
What has not been shown matters as much. We have not shown that the defect rate is 4.5% (the sample's rate): a 95% Wilson interval for it runs from about 2.4% to 8.3%. We have not computed the probability that the supplier is cheating. And the evidence is moderate, not overwhelming: a p-value of 0.02 is the kind of result that appears now and then even when nothing is wrong.
A test for a mean, and its link to intervals
Most tests follow the same pattern: (estimate − hypothesised value) divided by a standard error, compared with a known distribution.
Example— Do the solar lanterns last 10 hours?
A manufacturer advertises that its solar lanterns give 10 hours of light on a full charge. A consumer group in Dhaka tests 20 lanterns and records a mean of hours with standard deviation . Test against .
The standard error is , so with degrees of freedom. The two-sided p-value is . The 95% confidence interval is hours, and the effect size is , half a standard deviation.
Notice that the confidence interval excludes 10 and the two-sided p-value is below 0.05. That is no coincidence. A 95% confidence interval contains exactly the hypothesised values that a two-sided test at level 0.05 would not reject. This duality is why many statisticians prefer to report the interval: it shows the test's verdict for every hypothesised value at once, and the size of the effect too.
One-sided or two-sided? The one-sided p-value for is half the two-sided one, 0.019. Halving the p-value is tempting, which is exactly why the choice must be made before seeing the data and justified by the question. A one-sided test is honest only if an effect in the other direction would lead to the same decision as no effect at all. A consumer group that would also report lanterns lasting much longer than advertised should use a two-sided test. Switching to one-sided after seeing which way the data fell doubles the real type I error rate.
Errors, significance and power
Every test decision can go wrong in two ways.
| true | false | |
|---|---|---|
| Reject | type I error (false positive), probability | correct, probability = power |
| Do not reject | correct | type II error (false negative), probability |
The significance level controls the first row's left cell: the test is built so that, if is true, it rejects with probability . The simulation below checks this, and shows how easily it is broken. It runs 10,000 experiments in which the null hypothesis is exactly true, then repeats them with a researcher who "peeks" at the data after every 10 observations and stops as soon as .
import numpy as np
from scipy import stats
rng = np.random.default_rng(21)
reps, n = 10_000, 100
# 10,000 experiments in which the null hypothesis is TRUE: the mean really is 0.
data = rng.normal(0.0, 1.0, size=(reps, n))
p = stats.ttest_1samp(data, 0.0, axis=1).pvalue
print(f"share of p-values below 0.05: {np.mean(p < 0.05):.4f} (the type I error rate)")
print("p-values under a true null, in ten bins of width 0.1:")
counts, _ = np.histogram(p, bins=10, range=(0, 1))
print(counts)
# Peeking: test after every 10 observations and stop as soon as p < 0.05.
looks = range(10, n + 1, 10)
stopped_early = np.zeros(reps, dtype=bool)
for m in looks:
p_m = stats.ttest_1samp(data[:, :m], 0.0, axis=1).pvalue
stopped_early |= p_m < 0.05
print(f"\nwith 10 peeks, share of experiments that ever reach p < 0.05: {np.mean(stopped_early):.4f}")With a single, pre-planned test, 5.2% of the true-null experiments give , as promised (within simulation error), and the p-values are spread evenly between 0 and 1: under a true null, a p-value is uniformly distributed. That is what "type I error rate " means. With ten peeks, 20% of the experiments reach significance at some point, four times the nominal rate, although every single test was computed correctly. This is one of the forking paths by which honest researchers produce false findings: choosing when to stop, which outcome to report, which subgroup to analyse or which outliers to drop, after seeing the data. Sequential designs that allow planned interim looks exist, with adjusted thresholds, and are standard in clinical trials; unplanned peeking is not one of them.
Power is the other side. A test with low power usually misses real effects, so a study should be designed to have enough power for the smallest effect worth detecting. Power grows with the true effect size, with the sample size, and with , and falls as the data get noisier. It can be computed by formula, or simply by simulating the study many times in a world where the effect is real, which works for any design.
import numpy as np
from scipy import stats
rng = np.random.default_rng(13)
def power_by_simulation(effect, n, alpha=0.05, reps=10_000):
"""Share of two-sided one-sample t-tests that reject when the true mean is `effect` SDs from 0."""
data = rng.normal(effect, 1.0, size=(reps, n))
return np.mean(stats.ttest_1samp(data, 0.0, axis=1).pvalue < alpha)
def power_by_formula(effect, n, alpha=0.05):
"""The same power from the noncentral t distribution."""
df = n - 1
crit = stats.t.ppf(1 - alpha / 2, df)
nc = effect * np.sqrt(n)
return stats.nct.sf(crit, df, nc) + stats.nct.cdf(-crit, df, nc)
print("power to detect a mean difference of 0.5 standard deviations (alpha = 0.05, two-sided)")
print(" n simulated formula")
for n in [10, 20, 34, 50, 100]:
print(f"{n:4} {power_by_simulation(0.5, n):11.3f} {power_by_formula(0.5, n):9.3f}")
print("\npower with n = 34 for different true effects")
for d in [0.0, 0.2, 0.35, 0.5, 0.8]:
print(f"effect {d:4}: {power_by_formula(d, 34):.3f}")What p-values are not
By the 2010s, misuse of p-values had become serious enough that the American Statistical Association did something unprecedented: in 2016 it issued a formal statement on them (Wasserstein and Lazar, 2016). Its six principles are worth reading in the original. In summary, and in our words: a p-value measures how incompatible the data are with a particular statistical model; it is not the probability that the hypothesis is true, nor the probability that the results arose "by chance"; scientific and policy decisions should not rest only on whether a p-value crosses a threshold; honest inference needs full reporting of every analysis performed; a p-value does not measure the size or importance of an effect; and on its own a p-value is weak evidence for or against a hypothesis.
The commonest misreadings, with their corrections:
- " means a 2% chance that is true." The p-value is computed assuming ; it is , not . Turning one into the other needs Bayes' theorem and a prior, as in Bayesian inference. When most hypotheses tested in a field are false leads, a large share of "significant" results can be false positives even at .
- "A small p-value means a large or important effect." With enough data, a trivially small effect gives a tiny p-value. A drug that lowers blood pressure by 0.3 mmHg in a trial of 200,000 people would be highly significant and clinically useless.
- " and tell different stories." They are nearly the same evidence. The 0.05 line is a convention, not a law of nature.
Statisticians agree on these points and disagree, sometimes sharply, about the remedy. One group proposed lowering the default threshold for claims of new discoveries to 0.005 (Benjamin et al., 2018). Others argued for abandoning the language of "statistical significance" altogether in favour of reporting estimates and intervals and interpreting them in context (Amrhein, Greenland and McShane, 2019). Many journals now require effect sizes and confidence intervals alongside any p-value. In fields that test sharply defined predictions, such as particle physics, very strict thresholds are the norm (the "five sigma" standard corresponds to a one-sided p-value of about ).
The stakes became visible in the replication crisis. In 2015 the Open Science Collaboration repeated 100 published psychology experiments as closely as possible: 97% of the originals had reported significant results, but only about 36% of the replications did, and the replication effects were on average about half as large (Open Science Collaboration, 2015). Similar projects in economics, cancer biology and other fields found varied but sobering rates. The causes were not mainly fraud. They were small samples, flexible analyses and a publication system that rewarded , exactly the mechanisms this lesson simulates. The response has been constructive: preregistration, registered reports, larger collaborative studies and open data have spread rapidly, and the conversation about evidence is healthier for it.
Not significant is not no effect
"The difference was not statistically significant" is often reported as "there was no difference". The simulation below studies a real effect of 0.3 standard deviations with samples of 20, and looks at what such studies report.
import numpy as np
from scipy import stats
rng = np.random.default_rng(99)
true_effect, n, reps = 0.3, 20, 10_000 # a real but modest effect, studied with small samples
data = rng.normal(true_effect, 1.0, size=(reps, n))
result = stats.ttest_1samp(data, 0.0, axis=1)
estimates = data.mean(axis=1)
significant = result.pvalue < 0.05
print(f"true effect {true_effect}; share of studies with p < 0.05 (the power): {significant.mean():.3f}")
print(f"average estimate, all studies: {estimates.mean():.3f}")
print(f"average estimate, significant ones only: {estimates[significant].mean():.3f}")
# One of the non-significant studies, reported properly with its confidence interval.
i = np.flatnonzero(~significant)[0]
ci = stats.ttest_1samp(data[i], 0.0).confidence_interval(0.95)
print(f"\nstudy {i}: estimate {estimates[i]:.2f}, p = {result.pvalue[i]:.2f}, "
f"95% CI ({ci.low:.2f}, {ci.high:.2f})")The effect is real in every one of the 10,000 studies, yet only 24% reach . A study like number 2, with estimate 0.35 and , has not shown that the effect is absent. Its confidence interval, from to , includes zero, but it also includes effects more than twice the true size. The honest summary is "the data are consistent with no effect and with a large one; this study was too small to tell". Absence of evidence is not evidence of absence.
The middle lines show a subtler danger. Averaged over all studies, the estimate is unbiased (0.299). Averaged over the significant studies only, it is 0.570, nearly double the truth, because with only the samples that happened to overestimate the effect clear the threshold. If journals publish mainly significant results, the published literature systematically exaggerates effects. Andrew Gelman and John Carlin call this a type M (magnitude) error (2014), and it explains much of the shrinkage seen in replications.
To show that an effect is absent, or at least too small to matter, you need a different design: decide in advance the smallest effect that would matter, then check whether the confidence interval lies entirely inside that margin. This is an equivalence test (Lakens, 2017), and it is how regulators judge whether a generic medicine is equivalent to the original.
Testing in professional practice
Plan before you look. Good practice fixes the hypotheses, the primary outcome, the test, the sample size (from a power calculation) and the stopping rule before any data are seen, and records them publicly. Clinical trials must be registered in advance, in registries such as ClinicalTrials.gov and the WHO's International Clinical Trials Registry Platform. In other sciences, preregistration and registered reports (journals accept a study on its design, before the results are known) play the same role.
Report the effect, its interval, and everything you tried. A p-value without an estimate hides the size of the effect; an analysis reported without the others that were run hides the forking paths. Testing many hypotheses at once needs its own corrections, the subject of Multiple testing and reproducibility.
Tests in industry. Technology companies run thousands of randomised experiments (A/B tests) a year to decide which version of a product to ship; the methods, and the same peeking problem, are covered in the Data Science elective's experiments in products. Manufacturers use tests in acceptance sampling, as in the zip example, and in control charts.
Software. In Python, scipy.stats has binomtest, ttest_1samp and dozens of other tests, each returning a statistic and a p-value (and, for many, a confidence interval). In R, shown here for comparison and not run:
# R, not run here
binom.test(9, 200, p = 0.02, alternative = "greater")
t.test(hours, mu = 10) # two-sided one-sample t-test
power.t.test(delta = 0.5, sd = 1, power = 0.8, type = "one.sample")History. In the 1920s Ronald Fisher, at the Rothamsted agricultural research station in England, made the p-value popular as a continuous measure of evidence against a null hypothesis. His famous illustration was a colleague, Muriel Bristol, who said she could taste whether milk had been poured into the cup before or after the tea; Fisher designed a test with eight cups, four of each kind, in which guessing would identify all four milk-first cups with probability only . Jerzy Neyman and Egon Pearson then framed testing as a decision rule with two kinds of error and a power to be maximised (1933). Modern practice is an uneasy hybrid of the two views, which is one root of the confusion this lesson tries to clear up.
Common misconceptions
Misconception
"The p-value is the probability that the null hypothesis is true." It is the probability of data at least this extreme if the null hypothesis is true. These can differ enormously, depending on how plausible the hypothesis was to begin with.
Misconception
"Not significant means no effect." It means the data could not rule out zero, which is also what happens when a real effect is studied with too little data. Report the interval, and use an equivalence test to show an effect is negligible.
Misconception
"Significant means important." Statistical significance measures how clearly the data depart from the null model, which depends heavily on sample size. Importance depends on the size of the effect and the context: a tiny effect can be significant, and a large one can fail to be.
Misconception
"If , a replication has a 99% chance of being significant." The probability of a significant replication depends on the true effect and the replication's power, and is often far lower; the significance filter also means the original estimate is likely to be too large.
Misconception
"It is fine to look at the data, then pick the test (or the side, or the stopping point) that gives the smallest p-value." Each choice made after seeing the data inflates the type I error rate, as the peeking simulation showed (20% instead of 5%). Choose by the design and the question, in advance.
Exercises
1. Hypothesis testing
Warm-upHumanA trial compares a new cough syrup with a placebo and reports a difference in mean recovery time with . Which statement is correct?
2. Hypothesis testing
Warm-upHumanA genetics lab tests 200 genes, one test each at . Suppose that in truth none of the genes is linked to the disease. How many genes would you expect to be declared significant anyway?
3. Hypothesis testing
CoreHumanIn the zip example, the null hypothesis is that the defect rate is 2%, and the inspector tests 200 zips with a one-sided test at . What is the smallest number of defects that would lead her to reject the null hypothesis?
Useful values for : , , , .
4. Hypothesis testing
CoreHumanA researcher planned a two-sided test of whether a new teaching app changes exam scores. The result is , with scores higher in the app group. She then reports a one-sided test ("the app improves scores"), , significant at 0.05. What is wrong?
5. Hypothesis testing
CoreHumanA small trial of a salt-reduction programme in 60 adults finds that systolic blood pressure fell by 2.8 mmHg more in the programme group than in the control group, with 95% confidence interval mmHg and . Which conclusion is best supported?
6. Hypothesis testing
CoreHumanUsing the power table from the lesson's simulation (two-sided one-sample -test, , true effect 0.5 standard deviations), what is the smallest sample size in the table that gives at least 80% power?
| 10 | 20 | 34 | 50 | 100 | |
|---|---|---|---|---|---|
| power | 0.293 | 0.565 | 0.808 | 0.934 | 0.999 |
7. Hypothesis testing
CoreComputationalWrite p_value_at_least(k, n, p0), the one-sided p-value for , using only math.comb (no SciPy). The starter makes a common mistake: it returns , the probability of the observed count alone, instead of the whole tail of results at least as extreme.
8. Hypothesis testing
StretchComputationalWrite simulated_power(effect, n, reps, seed, alpha=0.05). With rng = np.random.default_rng(seed), simulate reps studies, each a sample of n values from a normal distribution with mean effect and standard deviation 1. For each study run a two-sided one-sample -test of mean 0 (scipy.stats.ttest_1samp accepts axis=1 to test every row at once). Return, as a float, the share of studies with p-value below alpha.
With effect=0 you are measuring the type I error rate, which should be close to alpha.
Further reading and resources
- The ASA statement on p-values: context, process, and purpose · Ronald L. Wasserstein and Nicole A. Lazar · 2016The American Statistical Association's six principles on what p-values do and do not mean. Short and worth reading in full.
- Moving to a world beyond "p < 0.05" · Ronald L. Wasserstein, Allen L. Schirm and Nicole A. Lazar · 2019The editorial introducing a special issue of The American Statistician on statistical inference after the 2016 statement.
- Redefine statistical significance · Daniel J. Benjamin and 71 co-authors · 2018The proposal to lower the default threshold for new discoveries from 0.05 to 0.005.
- Scientists rise up against statistical significance · Valentin Amrhein, Sander Greenland and Blake McShane · 2019A comment in Nature arguing for retiring dichotomous "significant or not" thinking altogether.
- Estimating the reproducibility of psychological science · Open Science Collaboration · 2015A large, coordinated attempt to replicate 100 published psychology studies.
- Beyond power calculations: assessing Type S (sign) and Type M (magnitude) errors · Andrew Gelman and John Carlin · 2014Why significant results from underpowered studies overstate effects.
- Equivalence tests: a practical primer for t tests, correlations, and meta-analyses · Daniël Lakens · 2017How to test for the absence of a meaningful effect.
- Preregistration · Center for Open ScienceWhat preregistering a study involves, and why.