Skip to content
LumenaryAcademy
Foundations 60 min

Sampling distributions and the central limit theorem

Every statistic varies from sample to sample. How much it varies is the standard error; why its distribution is so often normal is the central limit theorem; and when that theorem is slow, the simulation will tell you.

Code tested · 5AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Another sample, another answer

A transport researcher in Jakarta wants to know how long the city's workers spend commuting. She cannot ask all of them, so she surveys a random sample of 50 and finds a mean one-way commute of 42.5 minutes. A colleague runs the same survey the following week with a different random 50 and gets 54.1. Neither of them made a mistake. Different samples contain different people, so they give different answers.

This is the central fact of statistical inference: a statistic computed from a sample is itself a random quantity. Had chance picked a different sample, the statistic would have been different. Before we can say what a sample tells us about the population (the subject of every remaining lesson in this level), we need to know how much a statistic varies from one possible sample to another. Is the answer from 50 people likely to be within 1 minute of the truth, or within 10?

In Samples and surveys you saw that random sampling avoids bias. This lesson asks the next question: how far from the truth can a random sample's answer wander by chance alone? The answer has two parts. The standard error measures the size of the wandering, and it shrinks in proportion to 1/n1/\sqrt{n}. The central limit theorem describes its shape, which for averages of many independent pieces is close to normal, whatever the shape of the population. Both can be proved, and both can be watched happening in a simulation, which is what we will do.

IntuitionH

Three distributions you must not confuse

Three different distributions are in play whenever we sample, and most confusion in statistics comes from mixing them up.

  1. The population distribution: how the quantity varies across everyone in the population. Commute times in our synthetic Jakarta population range from a few minutes to over two hours, with mean μ=48\mu = 48 minutes and standard deviation σ=24\sigma = 24 minutes. It is skewed to the right: a long tail of very long commutes.
  2. The sample distribution: how the quantity varies across the 50 people we actually asked. It is a rough, noisy picture of the population distribution, and it is the only one of the three we ever see.
  3. The sampling distribution of a statistic: how the statistic (here, the sample mean) would vary across all the samples of 50 we could have drawn. It is a thought experiment: imagine repeating the survey thousands of times and making a histogram of the thousands of means.
populationmean 48, sd 24one sample50 workers, mean 44.1samplemeansmean 48, sd (standard error) 3.4020406080100120140commute time (minutes)
Three distributions on one axis, for a synthetic population of commute times. Top: the population, skewed to the right. Middle: one random sample of 50 workers, one dot each, with its mean marked below the axis. Bottom: the sampling distribution of the mean of 50, which is far narrower and almost symmetric. Each row has its own vertical scale; the bottom curve is really about six times taller than the top one.

The figure shows the striking thing. The sampling distribution of the mean is centred in the same place as the population, but it is far narrower, and although the population is lopsided, the distribution of the mean is nearly a symmetric bell.

DefinitionsH

Definitions

Definition— parameter and statistic

A parameter is a number describing the population, such as the population mean μ\mu or standard deviation σ\sigma. It is fixed but usually unknown. A statistic is a number computed from a sample, such as the sample mean Xˉ=1n∑i=1nXi\bar X = \frac{1}{n}\sum_{i=1}^n X_i or the sample standard deviation ss. It is known once we have the data, and it varies from sample to sample.

Definition— sampling distribution

The sampling distribution of a statistic is its probability distribution over repeated random samples of the same size, drawn in the same way from the same population.

Definition— standard error

The standard error (SE) of a statistic is the standard deviation of its sampling distribution. Since σ\sigma is usually unknown, we estimate the standard error of a mean by s/ns/\sqrt{n}, the estimated standard error, and software usually reports this simply as "the standard error".

The basic facts about the sample mean follow from the rules for expectation and variance in Distributions.

Theorem— Mean and standard error of the sample mean

If X1,…,XnX_1, \ldots, X_n are independent, each with mean μ\mu and standard deviation σ\sigma, then E[Xˉ]=μandSD⁡(Xˉ)=σn.E[\bar X] = \mu \qquad \text{and} \qquad \operatorname{SD}(\bar X) = \frac{\sigma}{\sqrt n}.

Proof

By linearity of expectation, E[Xˉ]=1n∑iE[Xi]=1n⋅nμ=μE[\bar X] = \frac1n \sum_i E[X_i] = \frac1n \cdot n\mu = \mu. Because the XiX_i are independent, their variances add: Var⁡(∑iXi)=nσ2\operatorname{Var}\big(\sum_i X_i\big) = n\sigma^2. Scaling by 1n\frac1n multiplies the variance by 1n2\frac1{n^2}, so Var⁡(Xˉ)=nσ2n2=σ2n\operatorname{Var}(\bar X) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}, and the standard deviation is σ/n\sigma/\sqrt n.

∎

Two consequences shape all of applied statistics. First, the sample mean is right on average: it has no systematic tendency to over- or underestimate. Second, its precision improves only with the square root of the sample size. To halve the standard error you need four times as many observations; to gain one more decimal place of precision, a hundred times as many. Precision is expensive.

Notice also what the formula does not contain: the size of the population. When the population is much larger than the sample, sampling without replacement is practically the same as sampling independently, and a sample of 1,000 is as precise for a country of 400,000 as for a country of 1.4 billion. (The exact correction factor, (N−n)/(N−1)\sqrt{(N - n)/(N - 1)}, is about 0.9987 for n=1000n = 1000 and N=390 000N = 390\,000.)

ComputationC

A survey, ten thousand times

We cannot run the Jakarta survey ten thousand times, but we can build a synthetic population and survey it as often as we like. Here the population is a million commute times drawn from a right-skewed gamma distribution, and each survey samples 50, 200 or 800 of them without replacement.

python
import numpy as np

rng = np.random.default_rng(42)

# A synthetic population: one-way commute times (minutes) of 1,000,000 workers. Right-skewed.
population = rng.gamma(shape=4.0, scale=12.0, size=1_000_000)
mu, sigma = population.mean(), population.std()
print(f"population: mean {mu:.2f} min, sd {sigma:.2f} min")

# Five surveys of 50 workers each give five different sample means.
for i in range(5):
    sample = rng.choice(population, size=50, replace=False)
    print(f"survey {i + 1}: sample mean {sample.mean():.2f}")

# Ten thousand surveys at each sample size: how much does the sample mean vary?
print("\n   n   sd of 10,000 sample means   sigma/sqrt(n)")
for n in [10, 50, 200, 800]:
    means = np.array([rng.choice(population, size=n, replace=False).mean() for _ in range(10_000)])
    print(f"{n:4} {means.std():18.2f} {sigma / np.sqrt(n):18.2f}")

def text_histogram(values, lo, hi, bins=12, width=40):
    counts, edges = np.histogram(values, bins=bins, range=(lo, hi))
    for c, left in zip(counts, edges):
        print(f"{left:6.0f} " + "#" * round(width * c / counts.max()))

print("\nthe population (one bar per 10 minutes):")
text_histogram(population, 0, 120)
means50 = np.array([rng.choice(population, size=50, replace=False).mean() for _ in range(10_000)])
print("\n10,000 sample means for n = 50, on the same scale:")
text_histogram(means50, 0, 120)
print("\nthe same 10,000 sample means, zoomed in (one bar per 2 minutes):")
text_histogram(means50, 36, 60)
print(f"mean of the sample means: {means50.mean():.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Five surveys of 50 give five different means, scattered across about 12 minutes. Across ten thousand surveys, the standard deviation of the sample means matches σ/n\sigma/\sqrt n at every sample size: 7.56 against 7.59 for n=10n = 10, and 0.86 against 0.85 for n=800n = 800. The text histograms reproduce the figure. On the population's scale, the 10,000 means are crammed into two bars; zoomed in, they form a nearly symmetric bell around 48, even though the population itself is skewed.

ExplanationH

The central limit theorem

The standard error tells us the size of the sampling distribution. Its shape is the business of one of the most remarkable results in mathematics.

Theorem— Central limit theorem

Let X1,X2,…X_1, X_2, \ldots be independent and identically distributed with mean μ\mu and finite standard deviation σ>0\sigma > 0. Then for every real zz, P ⁣(Xˉn−μσ/n≤z)⟶Φ(z)as n→∞,P\!\left(\frac{\bar X_n - \mu}{\sigma/\sqrt n} \le z\right) \longrightarrow \Phi(z) \quad \text{as } n \to \infty, where Φ\Phi is the cdf of the standard normal distribution.

In words: the standardised sample mean has approximately a standard normal distribution when nn is large, whatever the shape of the population, provided its variance is finite. The proof uses characteristic functions and is given in Lumenary Math's Probability theory; here we watch it work.

Because a sum is just nn times a mean, the theorem also says that sums of many independent pieces are approximately normal. That is the story of the normal distribution in the previous lesson. And because a proportion is the mean of 0-or-1 variables, sample proportions are approximately normal too, with standard error p(1−p)/n\sqrt{p(1-p)/n}.

-2024n = 1-2024n = 4-2024n = 16-2024n = 64standardised sample mean (solid) against the standard normal curve (dashed)
The central limit theorem at work on a skewed population. Each panel shows the exact distribution of the standardised mean of n exponential draws (solid) against the standard normal curve (dashed). At n = 1 it is the exponential itself; by n = 16 it is close to the bell; at n = 64 a slight lean to the right remains.

Where the theorem is slow

The theorem is a statement about a limit. It does not say how large nn must be, and the answer depends on the population. A useful measure is the population's skewness γ\gamma, its average cubed standardised distance from the mean. The skewness of the sample mean is γ/n\gamma/\sqrt n, so a population with skewness 2 (the exponential) gives a mean with skewness 0.37 at n=30n = 30, while a population with skewness 33 needs n≈8400n \approx 8400 before its mean is as symmetric as that. The Berry–Esseen theorem makes this precise: the largest gap between the cdf of the standardised mean and Φ\Phi is at most a constant (known to be below one half) times E∣X−μ∣3/(σ3n)E|X - \mu|^3 / (\sigma^3 \sqrt n).

Simulation shows what this means in practice. For each population below we draw 10,000 samples, and count how often the sample mean lands more than 1.96 standard errors below or above μ\mu. If the sampling distribution were exactly normal, each tail would hold 2.5%.

python
import numpy as np

rng = np.random.default_rng(1)

# Three populations with known mean and standard deviation, from mildly to wildly skewed.
populations = {
    "exponential": (lambda size: rng.exponential(1.0, size), 1.0, 1.0),
    "lognormal(sigma=1.5)": (lambda size: rng.lognormal(0.0, 1.5, size),
                             np.exp(1.125), np.sqrt((np.exp(2.25) - 1) * np.exp(2.25))),
    "Pareto(alpha=2.5)": (lambda size: rng.pareto(2.5, size) + 1.0,
                          2.5 / 1.5, np.sqrt(2.5 / (1.5**2 * 0.5))),
}

def sample_means(draw, n, reps=10_000, chunk=1_000):
    """reps sample means of samples of size n, a chunk at a time to save memory."""
    return np.concatenate([draw((chunk, n)).mean(axis=1) for _ in range(reps // chunk)])

print("If the sample mean were normal, the columns would read 0.025, 0.025 and 0.050.")
print(f"{'population':21} {'n':>5} {'below mu-1.96SE':>16} {'above mu+1.96SE':>16} {'total':>6}")
for name, (draw, mu, sigma) in populations.items():
    for n in [5, 30, 200, 1000]:
        m = sample_means(draw, n)
        se = sigma / np.sqrt(n)
        below, above = np.mean(m < mu - 1.96 * se), np.mean(m > mu + 1.96 * se)
        print(f"{name:21} {n:5} {below:16.3f} {above:16.3f} {below + above:6.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The exponential population behaves well: by n=200n = 200 the tails are 2.0% and 3.0%, and by n=1000n = 1000 they are essentially exact. The lognormal with σ=1.5\sigma = 1.5 (skewness about 33, typical of incomes or insurance claims) is another story. Even at n=1000n = 1000 the lower tail holds 0.9% and the upper tail 4.0%: the sample mean is still clearly skewed. The total outside ±1.96\pm 1.96 SE is close to 5% at n=1000n = 1000, which can hide the problem, but anyone making a one-sided claim would be wrong by a factor of about two. The Pareto population, whose third moment is infinite, converges more slowly still. Its sample mean falls a little below μ\mu more often than not, because the rare enormous values that hold up the average are usually missing from a sample, and occasionally it overshoots by a lot.

The practical advice is simple. Look at the data, and if they are strongly skewed or have a few extreme values, check the normal approximation by simulation (as above) or use methods that do not depend on it, such as the bootstrap of the next lesson or a transformation such as taking logarithms.

ExplanationH

The law of large numbers

The central limit theorem describes the fluctuations of the sample mean around μ\mu. An older and simpler result says that those fluctuations die away.

Theorem— Weak law of large numbers

If X1,X2,…X_1, X_2, \ldots are independent with mean μ\mu and finite variance σ2\sigma^2, then for every ε>0\varepsilon > 0, P(∣Xˉn−μ∣≥ε)≤σ2nε2⟶0.P\big(|\bar X_n - \mu| \ge \varepsilon\big) \le \frac{\sigma^2}{n\varepsilon^2} \longrightarrow 0.

Proof

Chebyshev's inequality says that any random variable YY with mean mm and variance vv satisfies P(∣Y−m∣≥ε)≤v/ε2P(|Y - m| \ge \varepsilon) \le v/\varepsilon^2. Apply it to Y=XˉnY = \bar X_n, which has mean μ\mu and variance σ2/n\sigma^2/n.

∎

The law of large numbers is why casinos and insurers can plan: individual outcomes are unpredictable, but averages over many independent outcomes are not. It is also why simulation works: the frequency of an event in many simulated repetitions converges to its probability. Jacob Bernoulli proved the first version for coin-like trials in his Ars Conjectandi, published in 1713, eight years after his death.

python
import numpy as np

rng = np.random.default_rng(2024)
N = 1_000_000
checkpoints = [10, 100, 1_000, 10_000, 100_000, 1_000_000]

# Running averages of die rolls (mean 3.5) and of Cauchy draws (no mean at all).
die = rng.integers(1, 7, N)
cauchy = rng.standard_cauchy(N)
steps = np.arange(1, N + 1)
die_avg = np.cumsum(die) / steps
cauchy_avg = np.cumsum(cauchy) / steps

print(f"{'rolls':>9} {'average of die':>15} {'distance from 3.5':>18} {'average of Cauchy':>18}")
for n in checkpoints:
    print(f"{n:9} {die_avg[n - 1]:15.4f} {abs(die_avg[n - 1] - 3.5):18.4f} {cauchy_avg[n - 1]:18.4f}")

# The law of large numbers works by swamping, not by compensating.
sixes = np.cumsum(die == 6)
for n in [600, 6_000, 60_000, 600_000]:
    excess = sixes[n - 1] - n / 6
    print(f"after {n:7} rolls: {sixes[n - 1]:6} sixes, {excess:+7.0f} from n/6, share {sixes[n - 1] / n:.4f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The average of the die rolls closes in on 3.5: off by 0.26 after 100 rolls, by 0.0011 after a million. The Cauchy distribution, whose tails are so heavy that it has no mean at all, never settles: after a million draws its running average is still jumping between values like −3-3 and +3+3, because every so often a single draw is large enough to drag the whole average.

The second table answers a common misunderstanding. After 600,000 rolls the number of sixes is 336 below n/6n/6, a larger shortfall than after 600 rolls, and yet the proportion of sixes, 0.1661, is closer to 1/61/6. The law of large numbers does not work by the dice "making up" for a shortage of sixes. Dice have no memory. It works by swamping: the early deviation stays roughly where it was while the number of rolls grows, so its effect on the proportion shrinks.

ExplanationH

Standard errors in professional practice

Every serious estimate comes with a standard error. National statistics offices publish them beside their estimates. The United States Census Bureau's American Community Survey, for example, prints a margin of error next to each figure, and the UK's Office for National Statistics and Statistics Canada publish confidence intervals for their survey estimates. An opinion poll's "margin of error of 3 percentage points" is roughly two standard errors of a proportion near 50% from about 1,000 respondents: 1.960.25/1000≈0.0311.96\sqrt{0.25/1000} \approx 0.031.

Real surveys are not simple random samples. Household surveys usually sample villages or city blocks first and then households within them (cluster sampling), because visiting a thousand scattered addresses is expensive. People in the same cluster resemble each other, so the sample carries less independent information than its size suggests. Survey statisticians measure this with the design effect, the ratio of the actual variance to the variance a simple random sample of the same size would give; in household surveys, values of 1.5 or more are common. Using s/ns/\sqrt n for a clustered sample understates the uncertainty.

Dependence is the assumption that most often fails. Daily measurements from one sensor, repeated tests on the same patients, and pupils in the same school are not independent. Positive correlation between observations makes the true standard error larger than s/ns/\sqrt n, sometimes by a lot. Later lessons on time series and multilevel models deal with this properly.

Other statistics have sampling distributions too. The median, a correlation, a regression slope and a machine-learning model's accuracy on a test set all vary from sample to sample. Some have formulas for their standard errors (for normal data, the median's is about 1.25 σ/n1.25\,\sigma/\sqrt n); for others the standard error is found by resampling the data, the bootstrap idea introduced in the next lesson and developed in Resampling and simulation.

History. Abraham de Moivre found the normal approximation to the binomial in 1733; Pierre-Simon Laplace extended it to sums of general independent variables around 1810; Aleksandr Lyapunov gave the first rigorous general proof in 1901; and George Pólya named it the central limit theorem in 1920, because of its central place in probability. Francis Galton's "quincunx", a board on which balls bounce left or right off rows of pins and pile up into a bell shape, made the idea visible in the 1870s, and versions of it are still found in science museums.

Common misconceptionsH

Common misconceptions

Misconception

"With a large enough sample, the data become normal." The data have whatever distribution the population has; a larger sample just shows that shape more clearly. The central limit theorem is about the distribution of the mean (and other averages) across repeated samples.

Misconception

"n ≥ 30 is always enough for the normal approximation." It depends on the population. For mildly skewed data 30 is often fine; for heavily skewed data such as incomes or claim sizes, the lognormal simulation above shows clear skewness in the mean at n=1000n = 1000. For infinite-variance populations, no nn is enough.

Misconception

"The standard deviation and the standard error are the same thing." The standard deviation describes how individuals vary (σ=24\sigma = 24 minutes of commute); it does not shrink as you collect more data. The standard error describes how the estimate varies (24/50≈3.424/\sqrt{50} \approx 3.4 minutes for a sample of 50), and it shrinks as nn grows. Reporting one in place of the other makes results look either far more or far less precise than they are.

Misconception

"After a run of reds at roulette, black is due." This is the gambler's fallacy, a misreading of the law of large numbers. The wheel has no memory; the proportion of reds settles down because later spins swamp the early run, not because later spins compensate for it.

Misconception

"A bigger population needs a bigger sample." For a population much larger than the sample, precision depends on the sample size nn, not on the population size NN. A well-drawn random sample of 1,000 is about as precise in Iceland as in India. What does need a bigger sample is wanting precise results for subgroups: each region or age group needs its own adequate nn.

Exercises

Exercises

1. Sampling distributions and the central limit theorem

Warm-upHuman

Daily water use per household in a town near Pune has standard deviation σ=12\sigma = 12 litres. A random sample of 36 households is surveyed. What is the standard error of the sample mean, in litres?

2. Sampling distributions and the central limit theorem

CoreHuman

A pilot study of 150 patients estimates mean recovery time with a standard error of 4 days. Assuming the same population and random sampling, how many patients would the main study need for a standard error of 2 days?

3. Sampling distributions and the central limit theorem

CoreHuman

Household incomes in a region are strongly skewed to the right. A statistician draws a random sample of 2,000 households. According to the central limit theorem, which of these is approximately normal?

Choose one

4. Sampling distributions and the central limit theorem

CoreHuman

Two polling firms each interview a simple random sample of 1,000 adults: one in Iceland (about 300,000 adults) and one in India (about a billion adults). Both find that about 50% support a policy. How do the standard errors of their estimates compare?

Choose one

5. Sampling distributions and the central limit theorem

Warm-upHuman

A fair roulette wheel has come up red eight times in a row. Which statement is correct?

Choose one

6. Sampling distributions and the central limit theorem

CoreComputational

Write sample_means(population, n, reps, seed). It should create rng = np.random.default_rng(seed), draw reps random samples of size n with replacement from the array population, and return a NumPy array of the reps sample means.

The tests check that your sampling distribution is centred on the population mean, and that its standard deviation matches σ/n\sigma/\sqrt n.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Sampling distributions and the central limit theorem

StretchComputational

The lesson claims that the skewness of a sample mean is the population's skewness divided by n\sqrt n. Check it for the exponential distribution, whose skewness is 2.

  1. Write skewness(x), returning the sample skewness m3/m23/2m_3 / m_2^{3/2} as a float, where mkm_k is the average of (xi−xˉ)k(x_i - \bar x)^k.
  2. Write mean_skewness(n, reps, seed): with rng = np.random.default_rng(seed), draw reps samples of size n from the Exponential distribution with mean 1, and return the skewness of the reps sample means.

The theory predicts about 2/n2/\sqrt n: 2 for n=1n = 1, 1 for n=4n = 4, 0.5 for n=16n = 16.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Sampling distributions and the central limit theorem

Warm-upHuman

A random sample of 1,000 voters in Ghana finds 50% support for a proposal. Using 1.961.96 standard errors, what is the poll's margin of error, in percentage points, to the nearest whole number?

Further reading and resources