Sampling and statistical reasoning

Lens

MotivationH

Two and a half million wrong answers

In 1936 the American magazine The Literary Digest mailed questionnaires to about ten million people, asking how they would vote in the presidential election. About $2.4$ million replied — one of the largest polls ever taken — and the magazine predicted a comfortable win for Alf Landon. Franklin Roosevelt won in a landslide, with about $61\%$ of the vote. The young pollster George Gallup, using a far smaller sample chosen to represent the voting public, correctly predicted Roosevelt's victory.

How can a few thousand answers beat two million? Because how a sample is chosen matters far more than how big it is. The Digest's names came largely from telephone directories and lists of car owners — in 1936, mostly better-off households — and the people who bothered to reply were not typical either.

Today, statistical sampling measures unemployment, crop yields and public health in almost every country. India's National Sample Survey, set up in 1950 under the statistician Prasanta Chandra Mahalanobis, was one of the first to do this on a national scale. This lesson explains why careful sampling works, how to say how far to trust its answers, and how to think clearly about relationships between variables — including when a relationship is, and is not, evidence of a cause.

IntuitionH

One spoonful of soup

Analogy, not a formal construction

To check the salt in a big pot of soup, you do not drink the whole pot: you stir it well and taste one spoonful. Stirring makes every spoonful like every other, so one tells you about all of them — and a spoonful is enough for a large pot as well as a small one. But if you never stir, a spoonful from the top tells you nothing reliable, however large your spoon. Random sampling is the statistician's way of stirring. (This is an analogy: the mathematics behind it is the behaviour of random samples described below.)

There are two very different kinds of error in a sample estimate.

Chance error (sampling variability): different random samples give different answers, scattered around the truth. Chance errors are as likely to be too high as too low, and they shrink as the sample grows.
Bias (systematic error): the method of collecting data pushes every answer in the same direction — towards households with telephones, towards people with strong opinions. Bias does not shrink with sample size. A bigger biased sample only gives a more precise wrong answer.

DefinitionsH

Definitions

Definition— population, sample, parameter, statistic

The population is the whole group we want to know about; a sample is the part of it that we actually observe. A parameter is a number describing the population (the proportion $p$ of all households with a rainwater tank); a statistic is a number computed from the sample (the proportion $\hat p$ of sampled households with one), used to estimate the parameter.

Definition— simple random sample

In a simple random sample of size $n$ , every subset of $n$ members of the population is equally likely to be the sample.

Definition— bias

A method of sampling or measuring is biased if it tends to give estimates that are systematically too high or too low. Common sources: selection bias (some members are less likely to be included — people without telephones), non-response (those chosen do not all answer, and those who answer differ from those who do not), voluntary response (people choose themselves, and strong opinions are over-represented), and response bias (leading questions, or answers people are embarrassed to give).

Definition— sampling distribution and standard error

The sampling distribution of a statistic is the pattern of values it takes over all possible random samples. Its standard deviation is the statistic's standard error (SE). For the proportion $\hat p$ in a simple random sample of size $n$ , taken from a population much larger than $n$ ,

\mathrm{SE}(\hat p) \approx \sqrt{\frac{p(1 - p)}{n}}.

Definition— margin of error and confidence interval

A $95\%$ confidence interval for a proportion is $\hat p \pm 1.96\,\sqrt{\hat p(1 - \hat p)/n}$ ; the half-width $1.96\sqrt{\hat p(1-\hat p)/n}$ is the margin of error. " $95\%$ " describes the method: in the long run, about $95\%$ of intervals built this way from random samples contain the true $p$ .

Definition— normal distribution

The normal distribution with mean $\mu$ and standard deviation $\sigma$ is the bell-shaped curve $y = \dfrac{1}{\sigma\sqrt{2\pi}} e^{-(x - \mu)^2/(2\sigma^2)}$ . About $68\%$ of its area lies within $\sigma$ of $\mu$ , about $95\%$ within $1.96\,\sigma$ , and about $99.7\%$ within $3\sigma$ . It is a model: many sampling distributions are close to normal, which is where the number $1.96$ comes from.

For paired data $(x_1, y_1), \dots, (x_n, y_n)$ , write $\bar x, \bar y$ for the means and

S_{xx} = \sum_i (x_i - \bar x)^2, \qquad S_{yy} = \sum_i (y_i - \bar y)^2, \qquad S_{xy} = \sum_i (x_i - \bar x)(y_i - \bar y).

Definition— least-squares line and correlation coefficient

The residual of the point $(x_i, y_i)$ from the line $y = a + bx$ is $y_i - (a + bx_i)$ . The least-squares line is the line that makes the sum of squared residuals as small as possible. The correlation coefficient is

r = \frac{S_{xy}}{\sqrt{S_{xx}\,S_{yy}}}

(defined when $S_{xx} > 0$ and $S_{yy} > 0$ ). It measures the strength and direction of a linear association.

Worked examplesH

Worked examples

Example— Spotting bias

A radio station asks listeners to text in whether they support a new stadium: voluntary response — people with strong feelings reply.
A survey about public transport is carried out at a car park: selection bias — people who take the bus are left out.
"Do you agree that the dangerous old bridge should be replaced?" — a leading question (response bias).
A health survey reaches only the people who are at home in the afternoon: selection and non-response bias — workers and students are missed.

Example— A margin of error

A water agency surveys a simple random sample of $400$ households in a town of $20\,000$ ; $146$ of them, $\hat p = 0.365$ , collect rainwater in a tank. The standard error is about $\sqrt{0.365 \times 0.635 / 400} \approx 0.024$ , so the margin of error is $1.96 \times 0.024 \approx 0.047$ , and the $95\%$ confidence interval runs from about $0.318$ to $0.412$ . The agency can report "about $37\%$ , give or take $5$ percentage points". (The sample is only $2\%$ of the town, so the approximation "population much larger than $n$ " is reasonable.)

Example— Fitting a line

Five students record the hours $x$ they spent revising and their quiz score $y$ out of $10$ :

$x$ (hours)	1	2	3	4	5
$y$ (score)	2	4	5	4	5

Here $\bar x = 3$ , $\bar y = 4$ ; the deviations are $-2, -1, 0, 1, 2$ and $-2, 0, 1, 0, 1$ , so $S_{xx} = 10$ , $S_{xy} = 4 + 0 + 0 + 0 + 2 = 6$ and $S_{yy} = 6$ . As we prove below, the least-squares line has slope $\hat b = S_{xy}/S_{xx} = 0.6$ and intercept $\hat a = \bar y - \hat b \bar x = 2.2$ :

\hat y = 2.2 + 0.6x, \qquad r = \frac{6}{\sqrt{10 \times 6}} \approx 0.77.

The residuals $-0.8, 0.6, 1.0, -0.6, -0.2$ add up to $0$ , and their squares add up to $2.4$ . With five students, of course, this describes these five and nothing more.

Example— A non-example: r = 0, yet a perfect relationship

For the points $x = -3, -2, \dots, 3$ with $y = x^2$ , $S_{xy} = \sum x_i^3 - \bar y \sum x_i = 0$ , so $r = 0$ . Yet $y$ is completely determined by $x$ . The correlation coefficient detects only linear association; $r = 0$ means "no linear association", not "no relationship". Always plot the data.

ProofH

What can be proved, and how

Statistics rests on two kinds of argument: probability, which describes how random samples behave, and algebra, which describes the summaries we compute. Both can be proved. Whether a particular sample was really random, or a particular cause is real, is a question about the world, which no theorem can settle.

Theorem— Theorem 1 (the mean is the least-squares centre)

For numbers $x_1, \dots, x_n$ with mean $\bar x$ and any number $c$ ,

\sum_i (x_i - c)^2 = \sum_i (x_i - \bar x)^2 + n(\bar x - c)^2.

In particular $\sum_i (x_i - c)^2$ is smallest exactly when $c = \bar x$ .

Proof

First, the deviations from the mean add up to zero: $\sum_i (x_i - \bar x) = \sum_i x_i - n\bar x = 0$ . Now write $x_i - c = (x_i - \bar x) + (\bar x - c)$ and expand the square:

\sum_i (x_i - c)^2 = \sum_i (x_i - \bar x)^2 + 2(\bar x - c)\sum_i (x_i - \bar x) + n(\bar x - c)^2.

The middle term is $0$ . The last term is never negative, and it is $0$ only when $c = \bar x$ .

∎

Theorem— Theorem 2 (the 1/√n law)

Suppose $X_1, \dots, X_n$ are independent measurements, each with mean $\mu$ and standard deviation $\sigma$ (for example, values drawn at random, with replacement, from a population). Then their average $\bar X = (X_1 + \cdots + X_n)/n$ has mean $\mu$ and standard deviation $\sigma / \sqrt{n}$ .

Proof

This uses two facts proved in probability theory: the variance (the square of the standard deviation) of a sum of independent random quantities is the sum of their variances, and multiplying a random quantity by a constant $c$ multiplies its variance by $c^2$ (and its mean by $c$ ). So $\operatorname{Var}(X_1 + \cdots + X_n) = n\sigma^2$ , and

\operatorname{Var}(\bar X) = \frac{1}{n^2} \cdot n\sigma^2 = \frac{\sigma^2}{n}, \qquad \text{so the standard deviation of } \bar X \text{ is } \frac{\sigma}{\sqrt n}.

The mean of $\bar X$ is $\frac1n (\mu + \cdots + \mu) = \mu$ , because means always add, independent or not.

∎

A proportion is an average of $0$ s and $1$ s (one for each "yes"), with $\sigma = \sqrt{p(1-p)}$ ; this gives the standard error $\sqrt{p(1-p)/n}$ above. (Sampling without replacement makes the measurements slightly dependent and the spread slightly smaller; the difference is negligible when the sample is a small part of the population.) The law has a striking consequence: to halve the margin of error you need four times as many observations. And notice what is missing from the formula: the size of the population. A random sample of $1000$ is about as informative about a country of $50$ million as about a city of $500\,000$ — the spoonful of soup.

Where does $1.96$ come from? The central limit theorem, which you will meet in probability theory, says that for large $n$ the sampling distribution of $\bar X$ is approximately normal — whatever the shape of the population. So about $95\%$ of samples give a $\bar X$ within $1.96$ standard errors of $\mu$ ; and $\mu$ is within $1.96$ standard errors of $\bar X$ in exactly those samples. We state this theorem here without proof (Mathlib has a formal proof, shown in the formal section).

Theorem— Theorem 3 (the least-squares line)

Suppose the $x_i$ are not all equal, so $S_{xx} > 0$ . For every line $y = a + bx$ , the sum of squared residuals is

\mathrm{SSE}(a, b) = S_{xx}\Big(b - \frac{S_{xy}}{S_{xx}}\Big)^2 + n\big(\bar y - a - b\bar x\big)^2 + \Big(S_{yy} - \frac{S_{xy}^2}{S_{xx}}\Big).

Hence the least-squares line is $y = \hat a + \hat b x$ with

\hat b = \frac{S_{xy}}{S_{xx}}, \qquad \hat a = \bar y - \hat b\,\bar x,

it passes through the point of means $(\bar x, \bar y)$ , and its sum of squared residuals is $S_{yy} - S_{xy}^2/S_{xx}$ .

Proof

Centre the data: let $X_i = x_i - \bar x$ and $Y_i = y_i - \bar y$ , so that $\sum X_i = \sum Y_i = 0$ . Writing $d = \bar y - a - b\bar x$ , each residual is

y_i - (a + bx_i) = (Y_i - bX_i) + d.

Squaring and adding, the cross term is $2d\sum_i (Y_i - bX_i) = 2d(0 - b \cdot 0) = 0$ , so

\mathrm{SSE}(a, b) = \sum_i (Y_i - bX_i)^2 + nd^2 = S_{yy} - 2bS_{xy} + b^2 S_{xx} + nd^2.

Completing the square in $b$ , $S_{yy} - 2bS_{xy} + b^2S_{xx} = S_{xx}\big(b - S_{xy}/S_{xx}\big)^2 + S_{yy} - S_{xy}^2/S_{xx}$ , which is the formula. Since $S_{xx} > 0$ and $n > 0$ , both squared terms are at least $0$ , and both are $0$ exactly when $b = \hat b$ and $a = \bar y - \hat b \bar x$ . Then $\bar y = \hat a + \hat b \bar x$ : the line passes through $(\bar x, \bar y)$ .

∎

Theorem— Theorem 4 (correlation lies between −1 and 1)

If $S_{xx} > 0$ and $S_{yy} > 0$ , then $-1 \le r \le 1$ . Moreover, $S_{yy}(1 - r^2)$ is the least sum of squared residuals, so $r = \pm 1$ exactly when all the points lie on one straight line.

Proof

The least sum of squared residuals is $S_{yy} - S_{xy}^2/S_{xx}$ (Theorem 3), and

S_{yy} - \frac{S_{xy}^2}{S_{xx}} = S_{yy}\Big(1 - \frac{S_{xy}^2}{S_{xx}S_{yy}}\Big) = S_{yy}(1 - r^2).

A sum of squares cannot be negative, so $1 - r^2 \ge 0$ , that is $-1 \le r \le 1$ . And $r^2 = 1$ exactly when this sum is $0$ , that is, when every residual is $0$ and every point lies on the least-squares line. (Conversely, if all points lie on a line $y = a + bx$ , that line has $\mathrm{SSE} = 0$ , so the least possible value is $0$ .)

∎

The inequality $S_{xy}^2 \le S_{xx} S_{yy}$ hidden in this proof is the Cauchy–Schwarz inequality $\big(\sum f_i g_i\big)^2 \le \big(\sum f_i^2\big)\big(\sum g_i^2\big)$ , applied to $f_i = x_i - \bar x$ and $g_i = y_i - \bar y$ . The number $r^2$ is often described as "the fraction of the variation in $y$ explained by the line": the line reduces the sum of squares from $S_{yy}$ (using $\bar y$ alone, Theorem 1) to $S_{yy}(1 - r^2)$ . In the revision example, $r^2 = 0.6$ and the sum of squares falls from $6$ to $2.4$ .

ExplanationH

Correlation and causation

Across many towns, ice-cream sales and the number of people rescued from the sea rise and fall together. Ice cream does not cause drowning: hot weather drives both. A third variable that influences both of the variables we are looking at is a confounder. Other ways for a correlation to arise without $x$ causing $y$ :

Reverse causation: towns with more police officers may have more crime — perhaps because more officers are sent where there is crime.
Chance: search enough pairs of variables and some will be strongly correlated by coincidence.
Selection: if only some cases are observed (only successful companies, only patients who reached hospital), relationships can appear that do not exist in the whole population.

The strongest evidence for causation comes from a randomised experiment: assign the treatment by chance — by lottery, not by choice — to some subjects and not to others. Randomisation balances confounders, including those nobody thought of, between the groups, so a clear difference in outcomes is evidence that the treatment caused it. Ronald Fisher made randomisation the foundation of experimental design in the 1920s and 1930s, and randomised controlled trials are now the standard test of new medicines.

When experiments are impossible or unethical, careful observational studies can still build a strong case. The link between smoking and lung cancer, established by Richard Doll, Austin Bradford Hill and others from 1950 onwards, rested on many studies: a strong association, a dose–response relationship (more cigarettes, more risk), the right order in time, consistency across countries, and a plausible biological mechanism.

Formal mathematicsF

The algebra of statistics, checked by Lean

Lean cannot check that a sample was random or that a cause is real. It can check the algebra on which the methods rest. Data are lists $x : \text{Fin } n \to \R$ , that is $x_0, \dots, x_{n-1}$ .

meanLean 4Formally verified

/-- The **mean** `x̄ = (x₁ + ⋯ + xₙ)/n` of `n` data values `x : Fin n → ℝ`. -/
noncomputable def mean {n : ℕ} (x : Fin n → ℝ) : ℝ := (∑ i, x i) / n