How can a thousand people speak for millions?
Before an election, a newspaper reports: "42% of voters plan to vote for the Harbour Party." (The party is made up; reports like this are not.) The country has 40 million voters. How many did the pollsters ask? Usually about one or two thousand.
That sounds impossible. How can a thousand people tell you what forty million think? And yet, done carefully, it works remarkably well, and it is how countries measure unemployment, health, prices, crime and much else, every month.
Here is a picture that helps. A biscuit factory bakes 10,000 biscuits an hour. To check that they are baked properly, an inspector does not break open all 10,000 (there would be nothing left to sell!). She picks 50, taken at random from all over the batch, and breaks those open. If the 50 are fine, she can be confident the batch is fine, as long as she picked them from all over, and not just the 50 on top that happened to be easiest to reach.
That "as long as" is what this lesson is about. A small sample can tell you a great deal about a huge population, but only if it is chosen in the right way. Choose it badly and even an enormous sample can be badly wrong, as you will see.
Populations, samples and censuses
Definition— population
The population is the whole group you want to know about: all the voters in a country, all the pupils in a city, all the biscuits baked today, all the fish in a lake.
Definition— sample
A sample is the part of the population you actually observe or ask. You use what you find in the sample to estimate facts about the population.
Definition— census
A census collects data from every member of the population, rather than from a sample.
The number you want to know, like "the percentage of all voters who support the Harbour Party", is a fact about the population. You almost never know it exactly. The number you can work out, like "the percentage of the 1,000 people we asked who support the Harbour Party", is a fact about the sample. The whole art of sampling is to make the second a good estimate of the first.
Why choosing at random works
Definition— simple random sample
A simple random sample is chosen so that every member of the population has the same chance of being picked, and every group of the same size is equally likely to be the sample. It is like putting every name in a giant hat, mixing well, and drawing names without looking.
Why is chance such a good way to choose? Because chance has no favourites. If the pollster chooses people, she might, without meaning to, pick people who look friendly, who live nearby, or who are at home in the afternoon. Each of those choices tilts the sample. A lottery cannot tilt it: rich and poor, young and old, keen and bored all have the same chance of being drawn. So, apart from luck, the sample looks like the population.
And luck itself can be measured! That is what probability is for. We cannot say exactly how far a random sample's answer will be from the truth, but we can say how far it is likely to be, and the bigger the sample, the smaller that distance. That is the margin of error, which you will meet below.
Here is a surprise: for a random sample, the size of the population hardly matters. A random sample of 1,000 people is about as accurate for a country of 300 million as for a city of 1 million. What matters is the size of the sample, not the fraction of the population it covers. A cook can taste a well-stirred pot of soup with one spoonful, whether the pot holds one litre or a hundred.
In practice, pollsters and statistics offices use cleverer kinds of random sampling. They may divide a country into regions and sample randomly within each one, so that every region is represented (a stratified sample), or first choose villages at random and then households within them (a cluster sample), which is much cheaper when people are spread across a big country. P. C. Mahalanobis, whom you met in the first lesson, designed huge surveys of this kind for India. But the key ingredient is always the same: chance, not choice, decides who is in the sample.
Bias: when a sample leans one way
Definition— bias
A sampling method is biased if it tends to give answers that are wrong in the same direction, again and again: always too high, or always too low. Taking a bigger sample by the same method does not fix bias; it just gives you the wrong answer more confidently.
Here are the most common causes.
Example— Self-selection: the people who choose to answer
A website asks its readers: "Should the school day start an hour later? Click to vote!" Thousands vote, and 71% say yes. But who bothers to vote? Mostly pupils who feel strongly, and the keenest are those who want the change. People who choose themselves for a sample are called a self-selected or voluntary sample, and they are usually not like everyone else. Phone-in votes, online polls and comment sections are all self-selected.
Example— Convenience: whoever is easiest to ask
Hana wants to know how often pupils in her town use the public library, so she stands by the library door and asks people as they come in. Almost everyone she asks uses the library, of course! Asking whoever is easiest to reach, such as your friends, the people in the shopping centre, or the pupils in your own class, is called a convenience sample, and it is biased whenever the convenient people differ from the rest.
Example— Wording: how you ask changes the answer
Compare two questions about the same plan:
- "Should the town spend money on a new sports hall?"
- "Should the town waste even more of our money on a sports hall nobody asked for?"
The second is a leading question: it pushes people towards one answer. Wording effects can be subtle, too. People tend to agree with whatever a question suggests, and they tend to give answers that make them look good ("Did you brush your teeth twice yesterday?"). Good surveys test their questions carefully on small groups before using them.
Example— Non-response: the people who don't answer
A survey about working hours is sent to 2,000 randomly chosen adults. Only 600 reply. Who doesn't reply? Quite possibly the people who work the longest hours, because they are too busy! Even a perfectly random sample becomes biased if the people who don't respond differ from those who do. This is called non-response bias, and it is one of the biggest problems facing surveys today, because fewer and fewer people answer their phones or reply to letters. Pollsters try to correct for it by weighting: if young people are under-represented among those who replied, each young person's answer is counted a little more. Weighting helps, but only for the differences you know about.
The margin of error, by simulation
Even a perfectly random sample with no bias at all will not hit the truth exactly, because of luck: which particular people happened to be drawn. How big is that luck?
We can find out by doing what no real pollster can afford: run the same survey a thousand times. In the simulation below (you can run it yourself in the next section), 45% of a large population would say "yes". Each survey asks 1,000 people chosen at random and reports the percentage who said yes.
The results pile up around the truth, in the same hump shape you saw for heights. Most surveys are within 1 or 2 points of 45%; a few are 4 or 5 points away. And 94.7% of them are within 3 points. That is what a pollster means by "a margin of error of plus or minus 3 points".
Definition— margin of error
The margin of error of a random sample says how far its answer is likely to be from the truth because of luck in who was chosen. Usually it is the distance that 95 out of 100 such surveys would stay within.
There is a handy rule of thumb for surveys that report a percentage: the margin of error is about 1 divided by the square root of the sample size. (The square root of a number is the number that, multiplied by itself, gives it: the square root of 100 is 10.)
| Sample size | Square root | 1 ÷ square root | Margin of error |
|---|---|---|---|
| 100 | 10 | 0.1 | about ±10 points |
| 400 | 20 | 0.05 | about ±5 points |
| 1,000 | about 31.6 | about 0.032 | about ±3 points |
| 10,000 | 100 | 0.01 | about ±1 point |
Notice what the table says: to halve the margin of error you need four times as many people. Going from 1,000 to 4,000 people costs four times as much and shrinks the margin only from about 3 points to about 1.5. That is why so many polls stop at about 1,000.
Running surveys in Python
First, a big biased sample against a small random one. The program makes up a city of 50,000 pupils, about 45% of whom want the school day to start later. Then it runs a voluntary online poll, in which keen pupils are three times as likely to answer, and a random sample of 500. Press Run.
import numpy as np
rng = np.random.default_rng(1936)
# A made-up city with 50,000 pupils. True means "wants the school day to start later".
N = 50_000
wants_later = rng.random(N) < 0.45
print(f"The whole population: {wants_later.mean():.1%} want a later start")
print()
# A voluntary online poll: keen pupils are three times as likely to bother answering.
answers = np.where(wants_later, rng.random(N) < 0.30, rng.random(N) < 0.10)
print(f"Online poll: {answers.sum():,} replies, {wants_later[answers].mean():.1%} want a later start")
# A simple random sample: 500 pupils picked by lottery, and all of them answer.
chosen = rng.choice(N, size=500, replace=False)
print(f"Random sample: 500 pupils, {wants_later[chosen].mean():.1%} want a later start")The truth in this made-up city is 44.4%. The online poll collected 9,398 replies, almost twenty times as many as the random sample, and got 70.9%: wrong by more than 26 points. The random sample of just 500 got 44.8%, less than half a point away. Size cannot rescue a biased method.
Next, the thousand surveys from the figure, with a histogram drawn in characters:
import numpy as np
rng = np.random.default_rng(21)
true_share = 0.45 # 45% of a large population would say "yes"
# Run 1,000 separate surveys, each asking 1,000 people chosen at random.
yes_counts = (rng.random((1_000, 1_000)) < true_share).sum(axis=1)
surveys = yes_counts / 10 # each survey's result, in percent
print("Results of 1,000 surveys of 1,000 people each (each # is 4 surveys)")
for low in range(39, 51):
count = int(((surveys >= low) & (surveys < low + 1)).sum())
print(f"{low}-{low + 1}% | {'#' * (count // 4)} {count}")
within = (abs(surveys - 45) <= 3).mean()
print(f"Lowest result: {surveys.min()}% Highest: {surveys.max()}%")
print(f"Surveys within 3 points of the truth: {within:.1%}")Last, the rule of thumb, tested. For each sample size, the program runs 2,000 surveys and finds the distance from the truth that 95% of them stay within:
import numpy as np
rng = np.random.default_rng(1948)
true_share = 0.45
print("sample size 95% of surveys land within rule of thumb 1/sqrt(n)")
for n in [100, 400, 1_000, 4_000, 10_000]:
results = rng.binomial(n, true_share, size=2_000) / n * 100 # 2,000 surveys of n people
spread = np.percentile(abs(results - 45), 95)
print(f"{n:>11,} ±{spread:4.1f} points ±{100 / np.sqrt(n):4.1f} points")rng.binomial(n, true_share, ...) is a quick way of asking n random people and counting the yeses. The simulated margins (10.0, 4.8, 3.2, 1.5 and 0.9 points) agree closely with the rule of thumb (10.0, 5.0, 3.2, 1.6 and 1.0). You will see where the rule comes from in Sampling distributions and the central limit theorem, and how statisticians turn it into a confidence interval in Estimation and confidence intervals.
Famous polling failures, and what they teach
1936: the biggest poll, the biggest miss. In the United States presidential election of 1936, a popular magazine, the Literary Digest, mailed about 10 million straw-poll ballots. About 2.4 million came back, an enormous sample, and the magazine predicted that Alf Landon would beat President Franklin Roosevelt comfortably. Roosevelt won about 61% of the vote and 46 of the 48 states. Meanwhile, a young pollster named George Gallup, with a far smaller but more carefully chosen sample, correctly predicted Roosevelt's win. What went wrong? The Digest's mailing lists came largely from telephone directories, car registrations and its own subscribers, which in the middle of the Great Depression over-represented better-off people. And the people who chose to send their ballots back were not typical either: a study published in 1988 found that both the lists and the non-response tilted the result. The lesson, still true today: a huge biased sample is worse than a small random one.
1948: stopping too soon. In 1948 the main American polls predicted that Thomas Dewey would defeat President Harry Truman, and one Chicago newspaper printed the headline "Dewey Defeats Truman" before the votes were counted. Truman won. The pollsters had used quota sampling, in which interviewers were told how many people of each kind to ask (so many men, so many women, so many of each age) but chose the individuals themselves, which let in bias. Some polls also stopped asking weeks before the election, missing voters who changed their minds late. After 1948, serious pollsters moved towards random sampling.
2015: unrepresentative samples. Before the United Kingdom's general election of 2015, the polls showed the two largest parties roughly level. On the day, the Conservatives won by about six and a half percentage points and took a majority. An independent inquiry for the British Polling Council and the Market Research Society concluded that the main cause was that the polls' samples were unrepresentative: they contained too many supporters of one party and too few of the other, in ways that weighting did not fix.
Polls are still usually close, and pollsters learn from each miss. But every failure has the same root: the people who answered were not like the people who voted.
Censuses: asking everyone
If samples are so good, why do countries still hold a census, counting every person? Because some questions need detail that no sample can give: exactly how many people live in each small town or district, so that schools, clinics and voting areas can be planned. Rulers counted their people in ancient China, Egypt and Rome. The United States has held a census every ten years since 1790, and the United Nations recommends that every country hold one at least once a decade. India held one every ten years from 1881 to 2011; the next was delayed by the COVID-19 pandemic, and its first stage, listing every house, began in April 2026, with the population count to follow in 2027.
Censuses are not perfect either. Some people are missed, often the people who are hardest to reach. So statistics offices check their census with... a sample! After the count, they carry out a careful random survey of some areas and compare it with the census to estimate how many people were missed. Several countries, such as Denmark and Finland, no longer send questionnaires at all, and instead produce census figures from official registers of people and homes.
Census answers are personal, so they are protected by law: they may be used only to produce statistics, never to check up on one person. In the United States, for example, individual census records stay sealed for 72 years. How official statistics can be useful and keep people's data private is a big modern question, which you will meet in Privacy and official statistics.
Common misconceptions
Misconception
"A bigger sample is always better." Only if it is collected the same, unbiased way. The 2.4 million ballots of 1936 lost to a much smaller random sample. Fix the method first; then a bigger sample shrinks the margin of error.
Misconception
"A sample must include a good percentage of the population, say 10%, to be reliable." For a random sample, what matters is how many people you ask, not what fraction of the population that is. A thousand people give about the same margin of error for a town or a whole country.
Misconception
"The margin of error covers all the ways a poll can be wrong." It covers only the luck of random sampling. Bias from who answered, how the question was worded, or people changing their minds is extra, and can be much larger.
Misconception
"Random means careless or haphazard." In statistics, random sampling is a careful, planned procedure in which chance (a lottery, or a computer's random numbers) decides who is chosen. Picking "whoever comes along" is not random; it is convenience.
Exercises
Spot the bias, estimate the margin of error, and run some surveys of your own.
1. Samples and surveys
Warm-upHumanA health researcher in Ghana weighs 300 randomly chosen 12-year-olds from across the country, to estimate the average weight of all 12-year-olds in Ghana. What is the population?
2. Samples and surveys
CoreHumanA head teacher wants to know what the school's 900 pupils think of the canteen food. Which plan gives the least biased sample?
3. Samples and surveys
CoreHumanWhich of these surveys are likely to be biased? Choose every one that is.
4. Samples and surveys
CoreHumanA poll asks 2,500 randomly chosen people a yes-or-no question. Using the rule of thumb (margin of error ≈ 1 ÷ the square root of the sample size), what is the margin of error, in percentage points? (The square root of 2,500 is 50.)
5. Samples and surveys
StretchHumanA pollster wants a margin of error of about ±1 percentage point. Using the rule of thumb, how many randomly chosen people does she need to ask?
6. Samples and surveys
CoreHumanIn 1936 the Literary Digest collected about 2.4 million ballots and wrongly predicted the US election, while George Gallup used a much smaller sample and got it right. What is the main lesson?
7. Samples and surveys
CoreComputationalThe code makes up a town of 20,000 people; population holds True for each person who would say "yes". Replace each ... so that:
chosenis a random sample of 400 different people's positions: userng.choice(20_000, size=400, replace=False);sample_percentis the percentage of the chosen people who say yes;marginis the rule-of-thumb margin of error in percentage points,100 / np.sqrt(400).
The program then prints the sample's answer, give or take the margin, next to the truth.
8. Samples and surveys
StretchComputationalWrite two small functions using the rule of thumb:
margin(n)returns the margin of error, in percentage points, for a random sample ofnpeople:100 / sqrt(n);people_needed(m)returns how many people you need for a margin ofmpercentage points. (Work backwards: the square root of the sample size must be100 / m.)
Further reading and resources
- Why the 1936 Literary Digest Poll Failed · Peverill Squire · 1988A study of the most famous polling failure, using a survey taken soon afterwards to show that both the sample and the people who chose to reply were biased.
- British Polling CouncilThe association of British polling companies, which publishes rules on how polls must be reported and the inquiry into the 2015 election polls.
- United Nations Statistics DivisionIncluding the UN's principles and recommendations for population censuses around the world.
- Prasanta Chandra Mahalanobis, biography · MacTutor History of MathematicsThe statistician who showed how large, well-planned sample surveys could serve a whole country.