Another sample, another answer
A transport researcher in Jakarta wants to know how long the city's workers spend commuting. She cannot ask all of them, so she surveys a random sample of 50 and finds a mean one-way commute of 42.5 minutes. A colleague runs the same survey the following week with a different random 50 and gets 54.1. Neither of them made a mistake. Different samples contain different people, so they give different answers.
This is the central fact of statistical inference: a statistic computed from a sample is itself a random quantity. Had chance picked a different sample, the statistic would have been different. Before we can say what a sample tells us about the population (the subject of every remaining lesson in this level), we need to know how much a statistic varies from one possible sample to another. Is the answer from 50 people likely to be within 1 minute of the truth, or within 10?
In Samples and surveys you saw that random sampling avoids bias. This lesson asks the next question: how far from the truth can a random sample's answer wander by chance alone? The answer has two parts. The standard error measures the size of the wandering, and it shrinks in proportion to . The central limit theorem describes its shape, which for averages of many independent pieces is close to normal, whatever the shape of the population. Both can be proved, and both can be watched happening in a simulation, which is what we will do.
Three distributions you must not confuse
Three different distributions are in play whenever we sample, and most confusion in statistics comes from mixing them up.
- The population distribution: how the quantity varies across everyone in the population. Commute times in our synthetic Jakarta population range from a few minutes to over two hours, with mean minutes and standard deviation minutes. It is skewed to the right: a long tail of very long commutes.
- The sample distribution: how the quantity varies across the 50 people we actually asked. It is a rough, noisy picture of the population distribution, and it is the only one of the three we ever see.
- The sampling distribution of a statistic: how the statistic (here, the sample mean) would vary across all the samples of 50 we could have drawn. It is a thought experiment: imagine repeating the survey thousands of times and making a histogram of the thousands of means.
The figure shows the striking thing. The sampling distribution of the mean is centred in the same place as the population, but it is far narrower, and although the population is lopsided, the distribution of the mean is nearly a symmetric bell.
Definitions
Definition— parameter and statistic
A parameter is a number describing the population, such as the population mean or standard deviation . It is fixed but usually unknown. A statistic is a number computed from a sample, such as the sample mean or the sample standard deviation . It is known once we have the data, and it varies from sample to sample.
Definition— sampling distribution
The sampling distribution of a statistic is its probability distribution over repeated random samples of the same size, drawn in the same way from the same population.
Definition— standard error
The standard error (SE) of a statistic is the standard deviation of its sampling distribution. Since is usually unknown, we estimate the standard error of a mean by , the estimated standard error, and software usually reports this simply as "the standard error".
The basic facts about the sample mean follow from the rules for expectation and variance in Distributions.
Theorem— Mean and standard error of the sample mean
If are independent, each with mean and standard deviation , then
Proof
By linearity of expectation, . Because the are independent, their variances add: . Scaling by multiplies the variance by , so , and the standard deviation is .
∎
Two consequences shape all of applied statistics. First, the sample mean is right on average: it has no systematic tendency to over- or underestimate. Second, its precision improves only with the square root of the sample size. To halve the standard error you need four times as many observations; to gain one more decimal place of precision, a hundred times as many. Precision is expensive.
Notice also what the formula does not contain: the size of the population. When the population is much larger than the sample, sampling without replacement is practically the same as sampling independently, and a sample of 1,000 is as precise for a country of 400,000 as for a country of 1.4 billion. (The exact correction factor, , is about 0.9987 for and .)
A survey, ten thousand times
We cannot run the Jakarta survey ten thousand times, but we can build a synthetic population and survey it as often as we like. Here the population is a million commute times drawn from a right-skewed gamma distribution, and each survey samples 50, 200 or 800 of them without replacement.
import numpy as np
rng = np.random.default_rng(42)
# A synthetic population: one-way commute times (minutes) of 1,000,000 workers. Right-skewed.
population = rng.gamma(shape=4.0, scale=12.0, size=1_000_000)
mu, sigma = population.mean(), population.std()
print(f"population: mean {mu:.2f} min, sd {sigma:.2f} min")
# Five surveys of 50 workers each give five different sample means.
for i in range(5):
sample = rng.choice(population, size=50, replace=False)
print(f"survey {i + 1}: sample mean {sample.mean():.2f}")
# Ten thousand surveys at each sample size: how much does the sample mean vary?
print("\n n sd of 10,000 sample means sigma/sqrt(n)")
for n in [10, 50, 200, 800]:
means = np.array([rng.choice(population, size=n, replace=False).mean() for _ in range(10_000)])
print(f"{n:4} {means.std():18.2f} {sigma / np.sqrt(n):18.2f}")
def text_histogram(values, lo, hi, bins=12, width=40):
counts, edges = np.histogram(values, bins=bins, range=(lo, hi))
for c, left in zip(counts, edges):
print(f"{left:6.0f} " + "#" * round(width * c / counts.max()))
print("\nthe population (one bar per 10 minutes):")
text_histogram(population, 0, 120)
means50 = np.array([rng.choice(population, size=50, replace=False).mean() for _ in range(10_000)])
print("\n10,000 sample means for n = 50, on the same scale:")
text_histogram(means50, 0, 120)
print("\nthe same 10,000 sample means, zoomed in (one bar per 2 minutes):")
text_histogram(means50, 36, 60)
print(f"mean of the sample means: {means50.mean():.2f}")Five surveys of 50 give five different means, scattered across about 12 minutes. Across ten thousand surveys, the standard deviation of the sample means matches at every sample size: 7.56 against 7.59 for , and 0.86 against 0.85 for . The text histograms reproduce the figure. On the population's scale, the 10,000 means are crammed into two bars; zoomed in, they form a nearly symmetric bell around 48, even though the population itself is skewed.
The central limit theorem
The standard error tells us the size of the sampling distribution. Its shape is the business of one of the most remarkable results in mathematics.
Theorem— Central limit theorem
Let be independent and identically distributed with mean and finite standard deviation . Then for every real , where is the cdf of the standard normal distribution.
In words: the standardised sample mean has approximately a standard normal distribution when is large, whatever the shape of the population, provided its variance is finite. The proof uses characteristic functions and is given in Lumenary Math's Probability theory; here we watch it work.
Because a sum is just times a mean, the theorem also says that sums of many independent pieces are approximately normal. That is the story of the normal distribution in the previous lesson. And because a proportion is the mean of 0-or-1 variables, sample proportions are approximately normal too, with standard error .
Where the theorem is slow
The theorem is a statement about a limit. It does not say how large must be, and the answer depends on the population. A useful measure is the population's skewness , its average cubed standardised distance from the mean. The skewness of the sample mean is , so a population with skewness 2 (the exponential) gives a mean with skewness 0.37 at , while a population with skewness 33 needs before its mean is as symmetric as that. The Berry–Esseen theorem makes this precise: the largest gap between the cdf of the standardised mean and is at most a constant (known to be below one half) times .
Simulation shows what this means in practice. For each population below we draw 10,000 samples, and count how often the sample mean lands more than 1.96 standard errors below or above . If the sampling distribution were exactly normal, each tail would hold 2.5%.
import numpy as np
rng = np.random.default_rng(1)
# Three populations with known mean and standard deviation, from mildly to wildly skewed.
populations = {
"exponential": (lambda size: rng.exponential(1.0, size), 1.0, 1.0),
"lognormal(sigma=1.5)": (lambda size: rng.lognormal(0.0, 1.5, size),
np.exp(1.125), np.sqrt((np.exp(2.25) - 1) * np.exp(2.25))),
"Pareto(alpha=2.5)": (lambda size: rng.pareto(2.5, size) + 1.0,
2.5 / 1.5, np.sqrt(2.5 / (1.5**2 * 0.5))),
}
def sample_means(draw, n, reps=10_000, chunk=1_000):
"""reps sample means of samples of size n, a chunk at a time to save memory."""
return np.concatenate([draw((chunk, n)).mean(axis=1) for _ in range(reps // chunk)])
print("If the sample mean were normal, the columns would read 0.025, 0.025 and 0.050.")
print(f"{'population':21} {'n':>5} {'below mu-1.96SE':>16} {'above mu+1.96SE':>16} {'total':>6}")
for name, (draw, mu, sigma) in populations.items():
for n in [5, 30, 200, 1000]:
m = sample_means(draw, n)
se = sigma / np.sqrt(n)
below, above = np.mean(m < mu - 1.96 * se), np.mean(m > mu + 1.96 * se)
print(f"{name:21} {n:5} {below:16.3f} {above:16.3f} {below + above:6.3f}")The exponential population behaves well: by the tails are 2.0% and 3.0%, and by they are essentially exact. The lognormal with (skewness about 33, typical of incomes or insurance claims) is another story. Even at the lower tail holds 0.9% and the upper tail 4.0%: the sample mean is still clearly skewed. The total outside SE is close to 5% at , which can hide the problem, but anyone making a one-sided claim would be wrong by a factor of about two. The Pareto population, whose third moment is infinite, converges more slowly still. Its sample mean falls a little below more often than not, because the rare enormous values that hold up the average are usually missing from a sample, and occasionally it overshoots by a lot.
The practical advice is simple. Look at the data, and if they are strongly skewed or have a few extreme values, check the normal approximation by simulation (as above) or use methods that do not depend on it, such as the bootstrap of the next lesson or a transformation such as taking logarithms.
The law of large numbers
The central limit theorem describes the fluctuations of the sample mean around . An older and simpler result says that those fluctuations die away.
Theorem— Weak law of large numbers
If are independent with mean and finite variance , then for every ,
Proof
Chebyshev's inequality says that any random variable with mean and variance satisfies . Apply it to , which has mean and variance .
∎
The law of large numbers is why casinos and insurers can plan: individual outcomes are unpredictable, but averages over many independent outcomes are not. It is also why simulation works: the frequency of an event in many simulated repetitions converges to its probability. Jacob Bernoulli proved the first version for coin-like trials in his Ars Conjectandi, published in 1713, eight years after his death.
import numpy as np
rng = np.random.default_rng(2024)
N = 1_000_000
checkpoints = [10, 100, 1_000, 10_000, 100_000, 1_000_000]
# Running averages of die rolls (mean 3.5) and of Cauchy draws (no mean at all).
die = rng.integers(1, 7, N)
cauchy = rng.standard_cauchy(N)
steps = np.arange(1, N + 1)
die_avg = np.cumsum(die) / steps
cauchy_avg = np.cumsum(cauchy) / steps
print(f"{'rolls':>9} {'average of die':>15} {'distance from 3.5':>18} {'average of Cauchy':>18}")
for n in checkpoints:
print(f"{n:9} {die_avg[n - 1]:15.4f} {abs(die_avg[n - 1] - 3.5):18.4f} {cauchy_avg[n - 1]:18.4f}")
# The law of large numbers works by swamping, not by compensating.
sixes = np.cumsum(die == 6)
for n in [600, 6_000, 60_000, 600_000]:
excess = sixes[n - 1] - n / 6
print(f"after {n:7} rolls: {sixes[n - 1]:6} sixes, {excess:+7.0f} from n/6, share {sixes[n - 1] / n:.4f}")The average of the die rolls closes in on 3.5: off by 0.26 after 100 rolls, by 0.0011 after a million. The Cauchy distribution, whose tails are so heavy that it has no mean at all, never settles: after a million draws its running average is still jumping between values like and , because every so often a single draw is large enough to drag the whole average.
The second table answers a common misunderstanding. After 600,000 rolls the number of sixes is 336 below , a larger shortfall than after 600 rolls, and yet the proportion of sixes, 0.1661, is closer to . The law of large numbers does not work by the dice "making up" for a shortage of sixes. Dice have no memory. It works by swamping: the early deviation stays roughly where it was while the number of rolls grows, so its effect on the proportion shrinks.
Standard errors in professional practice
Every serious estimate comes with a standard error. National statistics offices publish them beside their estimates. The United States Census Bureau's American Community Survey, for example, prints a margin of error next to each figure, and the UK's Office for National Statistics and Statistics Canada publish confidence intervals for their survey estimates. An opinion poll's "margin of error of 3 percentage points" is roughly two standard errors of a proportion near 50% from about 1,000 respondents: .
Real surveys are not simple random samples. Household surveys usually sample villages or city blocks first and then households within them (cluster sampling), because visiting a thousand scattered addresses is expensive. People in the same cluster resemble each other, so the sample carries less independent information than its size suggests. Survey statisticians measure this with the design effect, the ratio of the actual variance to the variance a simple random sample of the same size would give; in household surveys, values of 1.5 or more are common. Using for a clustered sample understates the uncertainty.
Dependence is the assumption that most often fails. Daily measurements from one sensor, repeated tests on the same patients, and pupils in the same school are not independent. Positive correlation between observations makes the true standard error larger than , sometimes by a lot. Later lessons on time series and multilevel models deal with this properly.
Other statistics have sampling distributions too. The median, a correlation, a regression slope and a machine-learning model's accuracy on a test set all vary from sample to sample. Some have formulas for their standard errors (for normal data, the median's is about ); for others the standard error is found by resampling the data, the bootstrap idea introduced in the next lesson and developed in Resampling and simulation.
History. Abraham de Moivre found the normal approximation to the binomial in 1733; Pierre-Simon Laplace extended it to sums of general independent variables around 1810; Aleksandr Lyapunov gave the first rigorous general proof in 1901; and George Pólya named it the central limit theorem in 1920, because of its central place in probability. Francis Galton's "quincunx", a board on which balls bounce left or right off rows of pins and pile up into a bell shape, made the idea visible in the 1870s, and versions of it are still found in science museums.
Common misconceptions
Misconception
"With a large enough sample, the data become normal." The data have whatever distribution the population has; a larger sample just shows that shape more clearly. The central limit theorem is about the distribution of the mean (and other averages) across repeated samples.
Misconception
"n ≥ 30 is always enough for the normal approximation." It depends on the population. For mildly skewed data 30 is often fine; for heavily skewed data such as incomes or claim sizes, the lognormal simulation above shows clear skewness in the mean at . For infinite-variance populations, no is enough.
Misconception
"The standard deviation and the standard error are the same thing." The standard deviation describes how individuals vary ( minutes of commute); it does not shrink as you collect more data. The standard error describes how the estimate varies ( minutes for a sample of 50), and it shrinks as grows. Reporting one in place of the other makes results look either far more or far less precise than they are.
Misconception
"After a run of reds at roulette, black is due." This is the gambler's fallacy, a misreading of the law of large numbers. The wheel has no memory; the proportion of reds settles down because later spins swamp the early run, not because later spins compensate for it.
Misconception
"A bigger population needs a bigger sample." For a population much larger than the sample, precision depends on the sample size , not on the population size . A well-drawn random sample of 1,000 is about as precise in Iceland as in India. What does need a bigger sample is wanting precise results for subgroups: each region or age group needs its own adequate .
Exercises
1. Sampling distributions and the central limit theorem
Warm-upHumanDaily water use per household in a town near Pune has standard deviation litres. A random sample of 36 households is surveyed. What is the standard error of the sample mean, in litres?
2. Sampling distributions and the central limit theorem
CoreHumanA pilot study of 150 patients estimates mean recovery time with a standard error of 4 days. Assuming the same population and random sampling, how many patients would the main study need for a standard error of 2 days?
3. Sampling distributions and the central limit theorem
CoreHumanHousehold incomes in a region are strongly skewed to the right. A statistician draws a random sample of 2,000 households. According to the central limit theorem, which of these is approximately normal?
4. Sampling distributions and the central limit theorem
CoreHumanTwo polling firms each interview a simple random sample of 1,000 adults: one in Iceland (about 300,000 adults) and one in India (about a billion adults). Both find that about 50% support a policy. How do the standard errors of their estimates compare?
5. Sampling distributions and the central limit theorem
Warm-upHumanA fair roulette wheel has come up red eight times in a row. Which statement is correct?
6. Sampling distributions and the central limit theorem
CoreComputationalWrite sample_means(population, n, reps, seed). It should create rng = np.random.default_rng(seed), draw reps random samples of size n with replacement from the array population, and return a NumPy array of the reps sample means.
The tests check that your sampling distribution is centred on the population mean, and that its standard deviation matches .
7. Sampling distributions and the central limit theorem
StretchComputationalThe lesson claims that the skewness of a sample mean is the population's skewness divided by . Check it for the exponential distribution, whose skewness is 2.
- Write
skewness(x), returning the sample skewness as a float, where is the average of . - Write
mean_skewness(n, reps, seed): withrng = np.random.default_rng(seed), drawrepssamples of sizenfrom the Exponential distribution with mean 1, and return the skewness of therepssample means.
The theory predicts about : 2 for , 1 for , 0.5 for .
8. Sampling distributions and the central limit theorem
Warm-upHumanA random sample of 1,000 voters in Ghana finds 50% support for a proposal. Using standard errors, what is the poll's margin of error, in percentage points, to the nearest whole number?
Further reading and resources
- Seeing Theory · Brown UniversityInteractive visual introductions to probability and statistics, including the central limit theorem.
- OpenIntro Statistics · David Diez, Mine Çetinkaya-Rundel and Christopher BarrA free, openly licensed textbook with a clear chapter on sampling distributions.
- American Community Survey · United States Census BureauA large national survey that publishes a margin of error beside every estimate.
- SciPy: statistical functions (scipy.stats)