A number and how far to trust it
A health ministry in Malawi wants to know what fraction of rural households have a solar panel, to plan a programme of support. A survey of 400 randomly chosen households finds 92 with a panel: 23%. Should the ministry plan for 23%?
It should plan for "about 23%", and the whole art of this lesson is to say how much "about" covers. From the previous lesson we know that a different sample of 400 would have given a different percentage, typically differing by about two percentage points. So a responsible report says: 23%, with a 95% confidence interval from 19% to 27%. The interval tells the reader how precise the survey was. A survey of only 40 households that found 9 with panels (22.5%) would report almost the same percentage, but its interval would run from about 12% to 38%; a planner who saw only the point estimates could not tell the two surveys apart.
Estimation is the craft of turning data into a best guess (a point estimate) together with an honest statement of its uncertainty (an interval estimate). It raises three questions that this lesson answers in turn. What makes one recipe for a guess better than another? What exactly does "95% confidence" promise? And how many observations do we need for a given precision?
Recipes, targets and nets
An estimator is a recipe: a rule that turns any possible sample into a number. "Add up the values and divide by " is an estimator of the population mean; "sort the values and take the middle one" is another. Because samples vary, the recipe's output varies, and we judge a recipe by how its outputs behave over all the samples it might meet, its sampling distribution.
Two different things can go wrong, and the picture of a dartboard keeps them apart.
A confidence interval adds a second idea. Instead of one throw at the target, we cast a net: a range computed from the data, wide enough that the procedure catches the true value in a stated fraction of samples, such as 95%.
Analogy, not a formal construction
Think of a fisher who throws a net into a murky lake to catch one particular fish she cannot see. Her nets are made so that 95% of throws catch it. After a throw she cannot see whether this net holds the fish; she knows only that her method succeeds 95 times in 100. The "95%" belongs to her way of throwing, not to any particular net.
Definitions
Definition— estimand, estimator, estimate
The estimand is the quantity we want to know, such as a population mean or proportion , written generally as . An estimator is a function of the sample, and therefore a random variable. An estimate is the value it takes on the data actually observed.
Definition— bias, standard error and mean squared error
The bias of is ; the estimator is unbiased if this is 0 for every . Its standard error is . Its mean squared error is , the average squared distance from the truth; its square root is the RMSE.
Definition— consistency
An estimator is consistent if, as the sample size grows, it converges in probability to : the chance of missing by more than any fixed amount tends to 0.
Definition— confidence interval
A confidence interval for is a pair of statistics such that for every possible value of . The probability is over repeated samples; it is called the coverage of the procedure. The value , usually 95%, is the confidence level.
Theorem— Bias–variance decomposition
For any estimator, .
Proof
Write . Then . Square and take expectations: the cross term is , leaving .
∎
The theorem explains why "unbiased" is not the same as "best". An estimator with a little bias and much less variance can have a smaller total error. This trade-off runs through all of modern statistics and machine learning; Lumenary Math's Statistical inference treats estimators and maximum likelihood more formally.
Choosing an estimator: a simulation
Two classic choices show the ideas at work. First, the sample variance: should we divide the sum of squared deviations by or by ? Second, for the centre of a distribution, should we use the sample mean or the sample median? We simulate 100,000 samples and measure each estimator's bias, standard error and RMSE.
import numpy as np
rng = np.random.default_rng(3)
reps = 100_000
def report(name, estimates, truth):
bias = estimates.mean() - truth
sd = estimates.std()
rmse = np.sqrt(np.mean((estimates - truth) ** 2))
print(f"{name:34} bias {bias:+7.3f} sd {sd:6.3f} rmse {rmse:6.3f}")
# 1. Two estimators of a variance (true value 4) from samples of size 5.
x = rng.normal(10.0, 2.0, size=(reps, 5))
report("variance, divide by n - 1", x.var(axis=1, ddof=1), 4.0)
report("variance, divide by n", x.var(axis=1, ddof=0), 4.0)
# 2. Mean or median for the centre (true value 0) of samples of size 25?
normal = rng.normal(0.0, 1.0, size=(reps, 25))
heavy = rng.standard_t(3, size=(reps, 25)) # t with 3 df: occasional wild values
print()
report("normal data: sample mean", normal.mean(axis=1), 0.0)
report("normal data: sample median", np.median(normal, axis=1), 0.0)
report("heavy-tailed data: sample mean", heavy.mean(axis=1), 0.0)
report("heavy-tailed data: sample median", np.median(heavy, axis=1), 0.0)Dividing by (Bessel's correction) removes the bias: the average estimate is 4.004 against a true variance of 4, while dividing by is too low by 0.8 on average, exactly . The deviations are measured from the sample mean, which sits closer to the data than the true mean does, so they are systematically too small, and corrects for it.
For the centre, the answer depends on the data. With normal data the mean wins (standard error 0.201 against 0.249 for the median). With heavy-tailed data, simulated here from a distribution with 3 degrees of freedom, occasional extreme values throw the mean around, and the median wins clearly (0.274 against 0.341). No estimator is best for every population, which is one reason to look at the data before choosing.
Intervals for means and proportions
The mean: Student's t interval
If the data are a random sample from a normal population, then follows a distribution with degrees of freedom. It looks like the standard normal but has heavier tails, because itself is estimated and sometimes comes out too small. Solving for , where is the 0.975-quantile of , gives the 95% interval William Sealy Gosset worked this out in 1908 as a brewer at Guinness in Dublin, where experiments on barley and yeast had small samples. Guinness let him publish only under a pen name, so it is still called Student's . As grows, approaches 1.96; for it is 2.131, and for it is 2.776.
Example— Teff yields in the Ethiopian highlands
An agronomist records the yield of teff on 16 randomly chosen plots: mean tonnes per hectare, standard deviation . The standard error is , and for 15 degrees of freedom, so the 95% interval is Using 1.96 instead of 2.131 would give : slightly too narrow, and with the coverage loss we will measure below.
The interval assumes a random sample, independent observations and a normal population. The normality assumption matters less as grows, thanks to the central limit theorem, but as the previous lesson showed, "grows" can mean a lot for skewed data.
The proportion: Wald and Wilson
For a proportion, has standard error . Plugging in for gives the textbook Wald interval . For the Malawi survey, , the standard error is 0.021, and the interval is .
The Wald interval has a known flaw: when is near 0 or 1, its estimated standard error is too small, and at it collapses to the single point 0. A clinic that sees no side effects among 30 patients would report the interval , which claims certainty from 30 people. The Wilson interval (Edwin Bidwell Wilson, 1927) inverts the test of instead of plugging in , and behaves much better: For 92 of 400 it gives , nearly the same as Wald. For 0 of 30 it gives : zero side effects in 30 patients is consistent with a true rate as high as about 11%. A handy shortcut for zero events is the rule of three: with 0 events in independent trials, is an approximate 95% upper bound for the rate.
What 95% means, checked by simulation
The definition of a confidence interval is a promise about a procedure: in repeated samples, intervals built this way contain the true value 95% of the time. The figure shows the promise in action.
Once a particular interval has been computed, such as for teff, it either contains the true mean yield or it does not; we just do not know which. The 95% describes how often the method succeeds, and that is what makes it a statement we can check. A simulation can generate thousands of samples from a population whose truth we know and count how often the intervals succeed.
import numpy as np
from scipy import stats
rng = np.random.default_rng(8)
reps = 20_000
def coverage(draw, true_mean, n, use_t=True):
"""Fraction of 95% intervals mean +/- crit * s / sqrt(n) that contain the true mean."""
x = draw((reps, n))
m, s = x.mean(axis=1), x.std(axis=1, ddof=1)
crit = stats.t.ppf(0.975, df=n - 1) if use_t else 1.96
half = crit * s / np.sqrt(n)
return np.mean((m - half <= true_mean) & (true_mean <= m + half))
normal = lambda size: rng.normal(50.0, 10.0, size)
skewed = lambda size: rng.lognormal(0.0, 1.0, size) # true mean exp(0.5)
print("coverage of nominal 95% intervals for a mean")
print(" n normal data, t normal data, 1.96 skewed data, t")
for n in [5, 10, 40]:
print(f"{n:3} {coverage(normal, 50.0, n):16.3f} {coverage(normal, 50.0, n, use_t=False):19.3f}"
f" {coverage(skewed, np.exp(0.5), n):16.3f}")
# For a proportion the coverage can be computed exactly: add up P(X = k) over the k
# whose interval contains the true p.
def wald(k, n, z=1.96):
p = k / n
half = z * np.sqrt(p * (1 - p) / n)
return p - half, p + half
def wilson(k, n, z=1.96):
p = k / n
centre = (p + z**2 / (2 * n)) / (1 + z**2 / n)
half = z / (1 + z**2 / n) * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
return centre - half, centre + half
def exact_coverage(method, n, p):
k = np.arange(n + 1)
lo, hi = method(k, n)
return np.sum(stats.binom.pmf(k, n, p) * ((lo <= p) & (p <= hi)))
print("\nexact coverage of nominal 95% intervals for a proportion")
for n, p in [(50, 0.5), (50, 0.1), (50, 0.02), (500, 0.02)]:
print(f"n = {n:3}, p = {p:4}: Wald {exact_coverage(wald, n, p):.3f} Wilson {exact_coverage(wilson, n, p):.3f}")The table shows three things.
- The interval delivers what it promises when its assumptions hold. With normal data, coverage is 0.952 at every sample size, within simulation error of 0.95.
- Shortcuts cost coverage in small samples. Using 1.96 in place of gives 87.8% coverage at ; the interval claims 95% confidence while missing one time in eight.
- Broken assumptions cost coverage too. For lognormal data the interval covers only 82% of the time at and 89% at . Skewness makes the sample mean and move together (samples that happen to include a large value have both a high mean and a large ), and the symmetric interval cannot keep up.
For proportions, coverage can be computed exactly by adding up binomial probabilities. With and , the Wald interval covers the truth only 63.5% of the time, and even at it falls short, at 92.8%; the Wilson interval stays close to 95%. Brown, Cai and DasGupta's 2001 review recommends against the Wald interval for exactly these reasons. (No interval for a discrete count hits 95% exactly; coverage zigzags as changes, which is why Wilson's numbers run a little above or below.)
The bootstrap: intervals by resampling
Formulas like exist for a few statistics. What is the standard error of a median, a ratio, a trimmed mean, or a machine-learning model's accuracy? In 1979 Bradley Efron proposed a general answer, the bootstrap, named after the impossible feat of lifting yourself up by your own bootstraps.
The idea uses the plug-in principle. The sampling distribution depends on the unknown population. The sample is our best picture of that population. So we pretend the sample is the population and draw new samples from it: resample values with replacement, compute the statistic, and repeat thousands of times. The spread of these bootstrap statistics estimates the standard error, and their 2.5th and 97.5th percentiles give a percentile bootstrap interval.
import numpy as np
# Synthetic data: monthly mobile data use (GB) reported by 40 people. Skewed, with a few heavy users.
rng = np.random.default_rng(12)
data = np.round(rng.lognormal(mean=1.6, sigma=0.8, size=40), 1)
print("data:", np.sort(data))
print(f"sample mean {data.mean():.2f} GB, sample median {np.median(data):.2f} GB")
# The bootstrap: resample the data with replacement, recompute the statistic, repeat.
B = 10_000
resamples = rng.choice(data, size=(B, data.size), replace=True)
boot_medians = np.median(resamples, axis=1)
boot_means = resamples.mean(axis=1)
lo, hi = np.percentile(boot_medians, [2.5, 97.5])
print(f"bootstrap standard error of the median: {boot_medians.std(ddof=1):.2f} GB")
print(f"95% percentile bootstrap interval for the median: ({lo:.2f}, {hi:.2f}) GB")
# For the mean we can compare the bootstrap with the formula s / sqrt(n).
print(f"bootstrap SE of the mean {boot_means.std(ddof=1):.3f} formula s/sqrt(n) {data.std(ddof=1) / np.sqrt(data.size):.3f}")The 40 reported values are skewed: most people use 2 to 10 GB a month, and three use about 20. The bootstrap standard error of the median is 0.68 GB, and the 95% percentile interval for the population median runs from 3.55 to 6.60 GB. For the mean, where we can check, the bootstrap standard error (0.754) agrees closely with (0.772).
The bootstrap is powerful, and it is easy to trust it too much.
- It cannot create information. Every resample is built from the same 40 values. If the sample is small or unrepresentative, so is every resample. With , percentile intervals tend to be too narrow.
- Some statistics break it. The bootstrap fails for the maximum of a sample, and is unreliable for statistics that depend on a few extreme values.
- The resampling must copy the design. Observations that come in clusters (pupils within schools) or in time order must be resampled as clusters or blocks, not one by one.
Better versions exist. The BCa (bias-corrected and accelerated) interval adjusts the percentiles for bias and skewness, and is the default in SciPy's scipy.stats.bootstrap. Tim Hesterberg's 2015 article in the references is a careful guide to what works. Resampling and simulation returns to the bootstrap and its relatives in depth.
How many observations?
The margin of error of a 95% interval for a mean is about . Setting it equal to a target margin and solving for gives the planning formula rounded up. The proportion formula needs a guess at ; since is largest at , using 0.5 gives a sample size that is enough whatever the truth.
import math
from scipy import stats
def n_for_mean(sigma, margin, conf=0.95):
"""Sample size so that a conf-level interval for a mean has half-width about `margin`."""
z = stats.norm.ppf(0.5 + conf / 2)
return math.ceil((z * sigma / margin) ** 2)
def n_for_proportion(margin, p=0.5, conf=0.95):
"""Sample size for a proportion; p = 0.5 is the worst (largest) case."""
z = stats.norm.ppf(0.5 + conf / 2)
return math.ceil(z**2 * p * (1 - p) / margin**2)
print("teff yields, sigma about 0.30 t/ha:")
for margin in [0.10, 0.05, 0.025]:
print(f" margin +/-{margin:5.3f} t/ha needs n = {n_for_mean(0.30, margin)}")
print("a proportion, worst case p = 0.5:")
for margin in [0.05, 0.03, 0.01]:
print(f" margin +/-{margin:.2f} needs n = {n_for_proportion(margin)}")
print(" margin +/-0.03 at 99% confidence needs n =", n_for_proportion(0.03, conf=0.99))
print(" margin +/-0.03 if p is near 0.1 needs n =", n_for_proportion(0.03, p=0.1))Each halving of the margin quadruples the sample, so a margin of point costs nine times as many interviews as points (9,604 against 1,068). Raising the confidence level from 95% to 99% raises the sample size by about 73%. These are minimums. In practice the planner inflates them for the share of people expected not to respond, and for the design effect of a clustered sample, and gets from a pilot study or a previous survey, which is itself uncertain. When the aim is to detect an effect rather than to measure a quantity, sample size is planned through power instead, in the next lesson.
Intervals in professional practice
Report the interval, not just the estimate. Medical journals following the CONSORT reporting guidelines for clinical trials expect an effect estimate with its confidence interval, and national statistics offices publish intervals or margins of error with their survey estimates. An estimate without an interval cannot be judged.
Software. In Python, scipy.stats.t.interval and scipy.stats.binomtest(...).proportion_ci(method="wilson") give the intervals above, and scipy.stats.bootstrap does resampling. In R, the same intervals come from three built-in functions, shown here for comparison and not run:
# R, not run here
t.test(yield_t_ha)$conf.int # Student's t interval for a mean
prop.test(92, 400)$conf.int # Wilson interval (with a continuity correction by default)
binom.test(0, 30)$conf.int # Clopper-Pearson "exact" intervalOther kinds of interval. A Bayesian credible interval answers the question many people wrongly think a confidence interval answers: given the data and a prior, where is the parameter, with 95% probability? It needs a prior distribution, and it is the subject of Bayesian inference. A prediction interval covers a single future observation rather than a mean, and is much wider. A tolerance interval covers a stated fraction of the population. Mixing these up is a common and consequential error in engineering reports.
History. Jerzy Neyman, a Polish statistician working in London, set out the theory of confidence intervals in 1937, including the interpretation used here. Ronald Fisher proposed a rival "fiducial" theory that never won general acceptance, and the debate between them was famously bitter. Efron's bootstrap arrived in 1979, just as computers became cheap enough to make it practical.
Common misconceptions
Misconception
"There is a 95% probability that the true value lies in this interval." In the frequentist framework the true value is fixed, so a computed interval either contains it or not. The 95% is the success rate of the method over repeated samples. (A Bayesian credible interval does support a probability statement of this kind, under its prior.)
Misconception
"95% of the data fall inside the confidence interval." The interval for a mean is much narrower than the spread of individual values: it shrinks like , while the data do not. For teff, individual plots ranged far beyond . A range meant to cover individual values is a prediction or tolerance interval.
Misconception
"An unbiased estimator is always better than a biased one." Total error is bias squared plus variance. The divide-by- variance estimate had a smaller RMSE than the unbiased one, and many of the best modern methods (shrinkage, regularisation) deliberately accept a little bias for a large cut in variance.
Misconception
"The bootstrap makes small samples reliable." The bootstrap reuses the sample you have; it cannot reveal what the sample missed. Its intervals are approximations that improve with , and they are often too narrow for small samples.
Misconception
"A wider interval means a worse study." A wider interval means a less precise one, but an honest wide interval is far more useful than a falsely narrow one. Intervals that are too narrow, from shortcuts or broken assumptions, are the dangerous ones, because they invite overconfident decisions.
Exercises
1. Estimation and confidence intervals
Warm-upHumanFrom a random sample of 16 plots, a 95% confidence interval for the mean teff yield is tonnes per hectare. Which statement is correct?
2. Estimation and confidence intervals
CoreHumanA food scientist in Valencia measures the vitamin C content of 9 oranges from one grove: mean 20 mg per 100 g, sample standard deviation 6. The 0.975-quantile of the distribution with 8 degrees of freedom is 2.306. What is the 95% interval for the grove's mean?
3. Estimation and confidence intervals
Warm-upHumanAn estimator of a river's mean flow is biased upwards by 3 m³/s and has a standard error of 4 m³/s. What is its mean squared error, in (m³/s)²?
4. Estimation and confidence intervals
CoreHumanA national statistics office wants to estimate the share of adults with a bank account to within percentage points at 95% confidence, with no prior idea of the answer. Assuming simple random sampling and using , what is the minimum sample size?
5. Estimation and confidence intervals
CoreHumanIn a trial in Nairobi, 150 patients receive a new formulation of a medicine and none has a serious allergic reaction. Using the rule of three, what is an approximate 95% upper bound for the rate of serious reactions, in percent?
6. Estimation and confidence intervals
CoreComputationalThe starter computes the Wald interval for a proportion, which collapses to when . Rewrite proportion_interval(k, n, z=1.96) to return the Wilson interval from the lesson instead, as a tuple (lo, hi) clipped to :
7. Estimation and confidence intervals
StretchComputationalWrite bootstrap_ci(data, stat, B, seed, level=0.95), a percentile bootstrap interval:
- create
rng = np.random.default_rng(seed); Btimes, resampledatawith replacement (same size asdata) and apply the functionstat(for examplenp.median) to the resample;- return the
(1 - level)/2and(1 + level)/2quantiles of theBbootstrap statistics as a tuple of two floats.
The tests use 400 normal values, for which the theory predicts an interval for the mean of about .
8. Estimation and confidence intervals
CoreHumanIn which of these situations would a plain percentile bootstrap (resampling individual observations) be unreliable? Select all that apply.
Further reading and resources
- The probable error of a mean · Student (William Sealy Gosset) · 1908The paper that introduced the t distribution, written by a brewer at Guinness in Dublin.
- Outline of a theory of statistical estimation based on the classical theory of probability · Jerzy Neyman · 1937The paper that set out confidence intervals and their frequentist interpretation.
- Interval estimation for a binomial proportion · Lawrence D. Brown, T. Tony Cai and Anirban DasGupta · 2001Why the textbook (Wald) interval for a proportion behaves badly, and which intervals to use instead.
- Bootstrap methods: another look at the jackknife · Bradley Efron · 1979The paper that introduced the bootstrap.
- What teachers should know about the bootstrap · Tim C. Hesterberg · 2015A practical, honest guide to when bootstrap intervals work and when they do not.
- SciPy: scipy.stats.bootstrap