Skip to content
LumenaryAcademy
Start 50 min

Samples and surveys

How a thousand well-chosen people can speak for millions, why a million badly chosen ones cannot, what a margin of error really measures, and what famous polling failures teach.

Code tested · 5AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

How can a thousand people speak for millions?

Before an election, a newspaper reports: "42% of voters plan to vote for the Harbour Party." (The party is made up; reports like this are not.) The country has 40 million voters. How many did the pollsters ask? Usually about one or two thousand.

That sounds impossible. How can a thousand people tell you what forty million think? And yet, done carefully, it works remarkably well, and it is how countries measure unemployment, health, prices, crime and much else, every month.

Here is a picture that helps. A biscuit factory bakes 10,000 biscuits an hour. To check that they are baked properly, an inspector does not break open all 10,000 (there would be nothing left to sell!). She picks 50, taken at random from all over the batch, and breaks those open. If the 50 are fine, she can be confident the batch is fine, as long as she picked them from all over, and not just the 50 on top that happened to be easiest to reach.

That "as long as" is what this lesson is about. A small sample can tell you a great deal about a huge population, but only if it is chosen in the right way. Choose it badly and even an enormous sample can be badly wrong, as you will see.

DefinitionsH

Populations, samples and censuses

Definition— population

The population is the whole group you want to know about: all the voters in a country, all the pupils in a city, all the biscuits baked today, all the fish in a lake.

Definition— sample

A sample is the part of the population you actually observe or ask. You use what you find in the sample to estimate facts about the population.

Definition— census

A census collects data from every member of the population, rather than from a sample.

The number you want to know, like "the percentage of all voters who support the Harbour Party", is a fact about the population. You almost never know it exactly. The number you can work out, like "the percentage of the 1,000 people we asked who support the Harbour Party", is a fact about the sample. The whole art of sampling is to make the second a good estimate of the first.

IntuitionH

Why choosing at random works

Definition— simple random sample

A simple random sample is chosen so that every member of the population has the same chance of being picked, and every group of the same size is equally likely to be the sample. It is like putting every name in a giant hat, mixing well, and drawing names without looking.

Why is chance such a good way to choose? Because chance has no favourites. If the pollster chooses people, she might, without meaning to, pick people who look friendly, who live nearby, or who are at home in the afternoon. Each of those choices tilts the sample. A lottery cannot tilt it: rich and poor, young and old, keen and bored all have the same chance of being drawn. So, apart from luck, the sample looks like the population.

And luck itself can be measured! That is what probability is for. We cannot say exactly how far a random sample's answer will be from the truth, but we can say how far it is likely to be, and the bigger the sample, the smaller that distance. That is the margin of error, which you will meet below.

random sampleconvenience samplechance decides: no favouriteswhoever is easiest to reach
Two samples of 18 from a population of 180 (each circle is one person). Left: a random sample, spread across the whole population by chance. Right: a convenience sample, the 18 people who happened to be easiest to reach, all bunched in one corner. If people in that corner differ from the rest, as people living near each other often do, the right-hand sample gives a biased answer.

Here is a surprise: for a random sample, the size of the population hardly matters. A random sample of 1,000 people is about as accurate for a country of 300 million as for a city of 1 million. What matters is the size of the sample, not the fraction of the population it covers. A cook can taste a well-stirred pot of soup with one spoonful, whether the pot holds one litre or a hundred.

In practice, pollsters and statistics offices use cleverer kinds of random sampling. They may divide a country into regions and sample randomly within each one, so that every region is represented (a stratified sample), or first choose villages at random and then households within them (a cluster sample), which is much cheaper when people are spread across a big country. P. C. Mahalanobis, whom you met in the first lesson, designed huge surveys of this kind for India. But the key ingredient is always the same: chance, not choice, decides who is in the sample.

Worked examplesH

Bias: when a sample leans one way

Definition— bias

A sampling method is biased if it tends to give answers that are wrong in the same direction, again and again: always too high, or always too low. Taking a bigger sample by the same method does not fix bias; it just gives you the wrong answer more confidently.

Here are the most common causes.

Example— Self-selection: the people who choose to answer

A website asks its readers: "Should the school day start an hour later? Click to vote!" Thousands vote, and 71% say yes. But who bothers to vote? Mostly pupils who feel strongly, and the keenest are those who want the change. People who choose themselves for a sample are called a self-selected or voluntary sample, and they are usually not like everyone else. Phone-in votes, online polls and comment sections are all self-selected.

Example— Convenience: whoever is easiest to ask

Hana wants to know how often pupils in her town use the public library, so she stands by the library door and asks people as they come in. Almost everyone she asks uses the library, of course! Asking whoever is easiest to reach, such as your friends, the people in the shopping centre, or the pupils in your own class, is called a convenience sample, and it is biased whenever the convenient people differ from the rest.

Example— Wording: how you ask changes the answer

Compare two questions about the same plan:

  • "Should the town spend money on a new sports hall?"
  • "Should the town waste even more of our money on a sports hall nobody asked for?"

The second is a leading question: it pushes people towards one answer. Wording effects can be subtle, too. People tend to agree with whatever a question suggests, and they tend to give answers that make them look good ("Did you brush your teeth twice yesterday?"). Good surveys test their questions carefully on small groups before using them.

Example— Non-response: the people who don't answer

A survey about working hours is sent to 2,000 randomly chosen adults. Only 600 reply. Who doesn't reply? Quite possibly the people who work the longest hours, because they are too busy! Even a perfectly random sample becomes biased if the people who don't respond differ from those who do. This is called non-response bias, and it is one of the biggest problems facing surveys today, because fewer and fewer people answer their phones or reply to letters. Pollsters try to correct for it by weighting: if young people are under-represented among those who replied, each young person's answer is counted a little more. Weighting helps, but only for the differences you know about.

IntuitionH

The margin of error, by simulation

Even a perfectly random sample with no bias at all will not hit the truth exactly, because of luck: which particular people happened to be drawn. How big is that luck?

We can find out by doing what no real pollster can afford: run the same survey a thousand times. In the simulation below (you can run it yourself in the next section), 45% of a large population would say "yes". Each survey asks 1,000 people chosen at random and reports the percentage who said yes.

40%42%44%46%48%50%truth: 45%result of one survey
The results of 1,000 simulated surveys, each asking 1,000 random people, when the true answer is 45%. Most surveys land close to 45%; 94.7% of them land within 3 percentage points (the shaded band from 42% to 48%). No survey was more than 5 points away.

The results pile up around the truth, in the same hump shape you saw for heights. Most surveys are within 1 or 2 points of 45%; a few are 4 or 5 points away. And 94.7% of them are within 3 points. That is what a pollster means by "a margin of error of plus or minus 3 points".

Definition— margin of error

The margin of error of a random sample says how far its answer is likely to be from the truth because of luck in who was chosen. Usually it is the distance that 95 out of 100 such surveys would stay within.

There is a handy rule of thumb for surveys that report a percentage: the margin of error is about 1 divided by the square root of the sample size. (The square root of a number is the number that, multiplied by itself, gives it: the square root of 100 is 10.)

Sample sizeSquare root1 ÷ square rootMargin of error
100100.1about ±10 points
400200.05about ±5 points
1,000about 31.6about 0.032about ±3 points
10,0001000.01about ±1 point

Notice what the table says: to halve the margin of error you need four times as many people. Going from 1,000 to 4,000 people costs four times as much and shrinks the margin only from about 3 points to about 1.5. That is why so many polls stop at about 1,000.

ComputationC

Running surveys in Python

First, a big biased sample against a small random one. The program makes up a city of 50,000 pupils, about 45% of whom want the school day to start later. Then it runs a voluntary online poll, in which keen pupils are three times as likely to answer, and a random sample of 500. Press Run.

python
import numpy as np

rng = np.random.default_rng(1936)

# A made-up city with 50,000 pupils. True means "wants the school day to start later".
N = 50_000
wants_later = rng.random(N) < 0.45
print(f"The whole population: {wants_later.mean():.1%} want a later start")
print()

# A voluntary online poll: keen pupils are three times as likely to bother answering.
answers = np.where(wants_later, rng.random(N) < 0.30, rng.random(N) < 0.10)
print(f"Online poll: {answers.sum():,} replies, {wants_later[answers].mean():.1%} want a later start")

# A simple random sample: 500 pupils picked by lottery, and all of them answer.
chosen = rng.choice(N, size=500, replace=False)
print(f"Random sample: 500 pupils, {wants_later[chosen].mean():.1%} want a later start")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The truth in this made-up city is 44.4%. The online poll collected 9,398 replies, almost twenty times as many as the random sample, and got 70.9%: wrong by more than 26 points. The random sample of just 500 got 44.8%, less than half a point away. Size cannot rescue a biased method.

Next, the thousand surveys from the figure, with a histogram drawn in characters:

python
import numpy as np

rng = np.random.default_rng(21)
true_share = 0.45           # 45% of a large population would say "yes"

# Run 1,000 separate surveys, each asking 1,000 people chosen at random.
yes_counts = (rng.random((1_000, 1_000)) < true_share).sum(axis=1)
surveys = yes_counts / 10   # each survey's result, in percent

print("Results of 1,000 surveys of 1,000 people each (each # is 4 surveys)")
for low in range(39, 51):
    count = int(((surveys >= low) & (surveys < low + 1)).sum())
    print(f"{low}-{low + 1}% | {'#' * (count // 4)} {count}")
within = (abs(surveys - 45) <= 3).mean()
print(f"Lowest result: {surveys.min()}%   Highest: {surveys.max()}%")
print(f"Surveys within 3 points of the truth: {within:.1%}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Last, the rule of thumb, tested. For each sample size, the program runs 2,000 surveys and finds the distance from the truth that 95% of them stay within:

python
import numpy as np

rng = np.random.default_rng(1948)
true_share = 0.45

print("sample size   95% of surveys land within   rule of thumb 1/sqrt(n)")
for n in [100, 400, 1_000, 4_000, 10_000]:
    results = rng.binomial(n, true_share, size=2_000) / n * 100     # 2,000 surveys of n people
    spread = np.percentile(abs(results - 45), 95)
    print(f"{n:>11,}   ±{spread:4.1f} points                  ±{100 / np.sqrt(n):4.1f} points")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

rng.binomial(n, true_share, ...) is a quick way of asking n random people and counting the yeses. The simulated margins (10.0, 4.8, 3.2, 1.5 and 0.9 points) agree closely with the rule of thumb (10.0, 5.0, 3.2, 1.6 and 1.0). You will see where the rule comes from in Sampling distributions and the central limit theorem, and how statisticians turn it into a confidence interval in Estimation and confidence intervals.

History & global context

Famous polling failures, and what they teach

1936: the biggest poll, the biggest miss. In the United States presidential election of 1936, a popular magazine, the Literary Digest, mailed about 10 million straw-poll ballots. About 2.4 million came back, an enormous sample, and the magazine predicted that Alf Landon would beat President Franklin Roosevelt comfortably. Roosevelt won about 61% of the vote and 46 of the 48 states. Meanwhile, a young pollster named George Gallup, with a far smaller but more carefully chosen sample, correctly predicted Roosevelt's win. What went wrong? The Digest's mailing lists came largely from telephone directories, car registrations and its own subscribers, which in the middle of the Great Depression over-represented better-off people. And the people who chose to send their ballots back were not typical either: a study published in 1988 found that both the lists and the non-response tilted the result. The lesson, still true today: a huge biased sample is worse than a small random one.

1948: stopping too soon. In 1948 the main American polls predicted that Thomas Dewey would defeat President Harry Truman, and one Chicago newspaper printed the headline "Dewey Defeats Truman" before the votes were counted. Truman won. The pollsters had used quota sampling, in which interviewers were told how many people of each kind to ask (so many men, so many women, so many of each age) but chose the individuals themselves, which let in bias. Some polls also stopped asking weeks before the election, missing voters who changed their minds late. After 1948, serious pollsters moved towards random sampling.

2015: unrepresentative samples. Before the United Kingdom's general election of 2015, the polls showed the two largest parties roughly level. On the day, the Conservatives won by about six and a half percentage points and took a majority. An independent inquiry for the British Polling Council and the Market Research Society concluded that the main cause was that the polls' samples were unrepresentative: they contained too many supporters of one party and too few of the other, in ways that weighting did not fix.

Polls are still usually close, and pollsters learn from each miss. But every failure has the same root: the people who answered were not like the people who voted.

Censuses: asking everyone

If samples are so good, why do countries still hold a census, counting every person? Because some questions need detail that no sample can give: exactly how many people live in each small town or district, so that schools, clinics and voting areas can be planned. Rulers counted their people in ancient China, Egypt and Rome. The United States has held a census every ten years since 1790, and the United Nations recommends that every country hold one at least once a decade. India held one every ten years from 1881 to 2011; the next was delayed by the COVID-19 pandemic, and its first stage, listing every house, began in April 2026, with the population count to follow in 2027.

Censuses are not perfect either. Some people are missed, often the people who are hardest to reach. So statistics offices check their census with... a sample! After the count, they carry out a careful random survey of some areas and compare it with the census to estimate how many people were missed. Several countries, such as Denmark and Finland, no longer send questionnaires at all, and instead produce census figures from official registers of people and homes.

Census answers are personal, so they are protected by law: they may be used only to produce statistics, never to check up on one person. In the United States, for example, individual census records stay sealed for 72 years. How official statistics can be useful and keep people's data private is a big modern question, which you will meet in Privacy and official statistics.

Common misconceptionsH

Common misconceptions

Misconception

"A bigger sample is always better." Only if it is collected the same, unbiased way. The 2.4 million ballots of 1936 lost to a much smaller random sample. Fix the method first; then a bigger sample shrinks the margin of error.

Misconception

"A sample must include a good percentage of the population, say 10%, to be reliable." For a random sample, what matters is how many people you ask, not what fraction of the population that is. A thousand people give about the same margin of error for a town or a whole country.

Misconception

"The margin of error covers all the ways a poll can be wrong." It covers only the luck of random sampling. Bias from who answered, how the question was worded, or people changing their minds is extra, and can be much larger.

Misconception

"Random means careless or haphazard." In statistics, random sampling is a careful, planned procedure in which chance (a lottery, or a computer's random numbers) decides who is chosen. Picking "whoever comes along" is not random; it is convenience.

Exercises

Exercises

Spot the bias, estimate the margin of error, and run some surveys of your own.

1. Samples and surveys

Warm-upHuman

A health researcher in Ghana weighs 300 randomly chosen 12-year-olds from across the country, to estimate the average weight of all 12-year-olds in Ghana. What is the population?

Choose one

2. Samples and surveys

CoreHuman

A head teacher wants to know what the school's 900 pupils think of the canteen food. Which plan gives the least biased sample?

Choose one

3. Samples and surveys

CoreHuman

Which of these surveys are likely to be biased? Choose every one that is.

Choose all that apply

4. Samples and surveys

CoreHuman

A poll asks 2,500 randomly chosen people a yes-or-no question. Using the rule of thumb (margin of error ≈ 1 ÷ the square root of the sample size), what is the margin of error, in percentage points? (The square root of 2,500 is 50.)

5. Samples and surveys

StretchHuman

A pollster wants a margin of error of about ±1 percentage point. Using the rule of thumb, how many randomly chosen people does she need to ask?

6. Samples and surveys

CoreHuman

In 1936 the Literary Digest collected about 2.4 million ballots and wrongly predicted the US election, while George Gallup used a much smaller sample and got it right. What is the main lesson?

Choose one

7. Samples and surveys

CoreComputational

The code makes up a town of 20,000 people; population holds True for each person who would say "yes". Replace each ... so that:

  • chosen is a random sample of 400 different people's positions: use rng.choice(20_000, size=400, replace=False);
  • sample_percent is the percentage of the chosen people who say yes;
  • margin is the rule-of-thumb margin of error in percentage points, 100 / np.sqrt(400).

The program then prints the sample's answer, give or take the margin, next to the truth.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

8. Samples and surveys

StretchComputational

Write two small functions using the rule of thumb:

  • margin(n) returns the margin of error, in percentage points, for a random sample of n people: 100 / sqrt(n);
  • people_needed(m) returns how many people you need for a margin of m percentage points. (Work backwards: the square root of the sample size must be 100 / m.)
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources

  • Why the 1936 Literary Digest Poll Failed · Peverill Squire · 1988A study of the most famous polling failure, using a survey taken soon afterwards to show that both the sample and the people who chose to reply were biased.
  • British Polling CouncilThe association of British polling companies, which publishes rules on how polls must be reported and the inquiry into the 2015 election polls.
  • United Nations Statistics DivisionIncluding the UN's principles and recommendations for population censuses around the world.
  • Prasanta Chandra Mahalanobis, biography · MacTutor History of MathematicsThe statistician who showed how large, well-planned sample surveys could serve a whole country.