True numbers, false impressions
Here are three made-up headlines of a kind you might see any week:
- "Crime soars in our city!" (above a chart whose bars look three times taller this year)
- "Children who eat breakfast get better marks: so make your child eat breakfast!"
- "New test for a rare disease is 90% accurate: so if you test positive, you almost certainly have it."
Every number behind these headlines might be perfectly true, and yet each headline could leave you believing something false. Statistics can mislead without a single lie, through what is shown, what is left out, and what is assumed. Sometimes this is done on purpose, to sell something or win an argument; more often it is an honest mistake by someone in a hurry.
This lesson is a field guide to the most common traps. Once you know them, you will spot them everywhere, in news reports, advertisements, social media posts and even school projects. That is not a reason to distrust all statistics. It is a reason to ask good questions, so that you can trust the statistics that deserve it.
Graphs that stretch the truth
The truncated axis
At a school, 84% of pupils passed the swimming test in 2024 and 87% in 2025. A small improvement. Now look at two charts of exactly the same numbers.
The right-hand chart is not lying, exactly: the axis labels are there for anyone who reads them. But our eyes compare the heights of the bars, and in the right-hand chart the 2025 bar is four times taller, because its top is 4 points above 83 while the 2024 bar's top is only 1 point above. A three-point change looks like a revolution.
The rule for bar charts: the bars' heights stand for amounts, so the axis must start at zero. (Line graphs are different: they show changes, and an axis that doesn't start at zero can be fine there, as long as it is clearly labelled and the scale isn't chosen to exaggerate.)
The cherry-picked range
A chess club's membership has grown from about 100 to about 170 over twenty years, with a few ups and downs. Someone who wants to argue the club is failing can show only the years 2019 to 2022, when membership fell from 162 to 146.
Choosing a start and end that suit your argument is called cherry-picking. It works with any data that goes up and down: prices, temperatures, a sports team's results, a company's sales. The defence is simple: ask why does the chart start and stop where it does?, and look for a longer view.
Other graph tricks to watch for: pictures of objects (coins, people) drawn twice as tall and twice as wide, so they look four times as big; two lines drawn against two different axes so that they seem to move together; and charts with no labels on the axes at all.
Correlation is not causation
Definition— correlation
Two numerical variables are correlated when they tend to rise and fall together (or one tends to rise when the other falls). On a scatter plot, like the height and arm span plot in Describing data, correlated variables make a sloping cloud of dots rather than a shapeless blob.
Here is a true-sounding fact: in a seaside town, on days when more ice cream is sold, more people get sunburnt. Does ice cream cause sunburn? Of course not. Both are caused by something else, hot sunny weather, which makes people buy ice cream and makes people burn.
When two things are correlated, there are always several possible explanations:
- A causes B. Revising for a test raises your mark.
- B causes A (reverse causation). Fires attended by more fire engines turn out to do more damage, but it is the big fires that bring lots of fire engines, not the fire engines that make fires big.
- A hidden third thing causes both, a confounder. In a primary school, children with bigger feet read better. Big feet don't help reading: older children have bigger feet and have had more years of practice. Age is the confounder.
- Coincidence. Look at enough pairs of data series and some will move together by pure chance. The website Spurious Correlations collects real examples, such as yearly figures for two utterly unrelated things that rose and fell in step for a decade.
A serious example
For years, large studies that followed thousands of women found that those who took hormone therapy after the menopause had less heart disease than those who didn't. Many doctors recommended it partly for that reason. Then the Women's Health Initiative, a huge experiment in the United States in which women were assigned to the therapy or to a dummy pill at random, reported in 2002 that the therapy did not protect against heart disease at all, and slightly raised some risks. The earlier studies had been misled by a confounder: the women who chose hormone therapy tended to be healthier and better-off in the first place.
That is why the strongest evidence that A causes B comes from a randomised experiment: when chance decides who gets the treatment, as in Kofi's coin-tossing tomato test in What is statistics?, confounders are shared out evenly between the groups.
Correlation is still evidence, though. Nobody could ethically assign people at random to smoke for thirty years, yet in the 1950s careful studies, such as those of Richard Doll and Austin Bradford Hill in Britain, found that smokers developed lung cancer far more often than non-smokers. The link was very strong, appeared in study after study and country after country, grew with the number of cigarettes, and had no believable confounder. Together, that evidence convinced scientists that smoking causes lung cancer. Working out causes without an experiment is a whole science, which you will meet in Causal inference.
Medical tests and base rates
Here is one of the most important, and most misunderstood, calculations in everyday statistics.
A disease affects 1 in every 100 people. A test for it:
- correctly says "positive" for 90% of people who have the disease;
- wrongly says "positive" for 5% of people who don't have it (a false positive).
Saba takes the test, and it says positive. What is the chance that she really has the disease?
Most people, and many doctors when they are asked in a hurry, guess about 90%. The true answer is much lower. The clearest way to see why is to imagine 10,000 people taking the test, and count. Thinking with counts of people like this is called using natural frequencies.
Count the positives: 90 people who are ill, plus 495 who are healthy, makes 585 positive results. Of those 585, only 90 have the disease. So Saba's chance of having it is 90 out of 585, about 15%: worrying enough to do a second test, but far from certain.
Why so low? Because the disease is rare. Only 1 person in 100 has it, so the healthy people outnumber the ill ones 99 to 1, and even a small false-positive rate among so many healthy people produces more false alarms than true alarms. How common something is to begin with is called the base rate, and ignoring it is one of the commonest mistakes in reasoning about chance. (In Lumenary Math, this is the rule of conditional probability; natural frequencies give the same answer without any formulas.)
This is why doctors don't test everyone for every rare disease, and why a positive result from a screening test is usually followed by a second, different test. It is also why "90% accurate" is a phrase to be wary of: accurate for whom, and how common is the thing being tested for?
Averages of averages
Example— Two classes
Class A has 10 pupils, whose mean test mark is 80. Class B has 30 pupils, whose mean mark is 60. What is the mean mark of all 40 pupils?
The tempting answer is halfway between 80 and 60, which is 70. But that treats the two classes as if they were the same size. Count the marks instead: class A's marks add up to 10 × 80 = 800, and class B's to 30 × 60 = 1,800. Together, 2,600 marks shared among 40 pupils gives 2,600 ÷ 40 = 65. The bigger class pulls the overall mean towards its own.
That mistake is easy to fix once you know it. The next one is stranger.
Example— Two basketball players
Amara and Bea practise shooting, some shots close to the basket and some from far away.
| Close shots | Far shots | All shots | |
|---|---|---|---|
| Amara | 9 of 10 (90%) | 30 of 100 (30%) | 39 of 110 (35%) |
| Bea | 80 of 100 (80%) | 2 of 10 (20%) | 82 of 110 (75%) |
Amara is the better shooter from close up (90% against 80%) and the better shooter from far away (30% against 20%). Yet overall, Bea scores 75% of her shots and Amara only 35%! How?
Look at which shots each player took. Amara took most of her shots from far away, where everyone misses a lot; Bea took most of hers from close up, where everyone scores a lot. The overall percentages mostly measure who took the easy shots, not who shoots better.
When a comparison within every group points one way but the combined comparison points the other, it is called Simpson's paradox, after the British statistician Edward Simpson, who wrote about it in 1951. It turns up in real life with hospitals (one hospital may treat more of the most seriously ill patients), schools, universities and medicines. The cure is to ask: are the groups being compared made up of the same mix?
Checking claims in Python
Here is the medical test from above, with the base rate varied from 1 in 100 to 1 in 2. Press Run.
# A test that finds 90% of people who have a disease,
# but also wrongly says "positive" for 5% of people who don't.
def positives(people, base_rate, catches=0.90, false_alarm=0.05):
sick = people * base_rate
healthy = people - sick
true_pos = sick * catches
false_pos = healthy * false_alarm
return true_pos, false_pos
print("base rate sick & positive healthy & positive share of positives who are sick")
for rate in [0.01, 0.02, 0.10, 0.50]:
tp, fp = positives(10_000, rate)
print(f"{rate:9.0%} {tp:15.0f} {fp:18.0f} {tp / (tp + fp):10.0%}")The same test means very different things in different groups of people. Try making the test better at avoiding false alarms (false_alarm=0.01), and see how much the first row improves.
Next, Simpson's paradox, computed from the shot counts:
# Two basketball players' shots, from close to the basket and from far away.
shots = {
"Amara": {"close": (9, 10), "far": (30, 100)}, # (shots scored, shots taken)
"Bea": {"close": (80, 100), "far": (2, 10)},
}
for player, record in shots.items():
for distance, (scored, taken) in record.items():
print(f"{player:5} {distance:5}: {scored:3} of {taken:3} = {scored / taken:.0%}")
total_scored = sum(s for s, t in record.values())
total_taken = sum(t for s, t in record.values())
print(f"{player:5} overall: {total_scored:3} of {total_taken:3} = {total_scored / total_taken:.0%}")
print()Last, a warning about looking at too many things at once. The program invents 200 people, each of whom either ate or didn't eat each of 20 foods, and either had a bad dream or didn't, all completely at random: no food has anything to do with dreams. Then, for each food, it compares the percentage of eaters who had a bad dream with the percentage of non-eaters who did.
import numpy as np
rng = np.random.default_rng(20)
people, foods = 200, 20
# 200 made-up people. Each one ate (True) or didn't eat (False) each of 20 foods,
# and had a bad dream or not -- all completely at random, with NO link at all.
ate = rng.random((people, foods)) < 0.5
bad_dream = rng.random(people) < 0.3
differences = []
for f in range(foods):
eaters = bad_dream[ate[:, f]].mean()
others = bad_dream[~ate[:, f]].mean()
differences.append(eaters - others)
differences = np.array(differences)
worst = int(np.argmax(np.abs(differences)))
print("Difference in bad dreams (eaters minus non-eaters), food by food, in percentage points:")
print(np.round(differences * 100).astype(int).tolist())
print(f"Biggest: food number {worst + 1}, a difference of {differences[worst] * 100:+.0f} percentage points")Most differences are small, but food number 17 shows a difference of 14 percentage points: eaters of food 17 had bad dreams noticeably more often. A careless report could announce "Food 17 linked to nightmares!" But we know there is no link at all, because we made the data up that way. Look at twenty foods and one of them will very likely look interesting by pure chance.
How to read a headline about a study
When a news story says "Scientists say...", here are the questions a statistician asks. You don't need to ask all of them every time; even two or three will protect you from most traps.
- Who or what was studied? People, mice, or cells in a dish? How many? A result in 20 mice is a reason for more research, not a reason to change what you eat.
- Was it an experiment or an observation? If people chose for themselves (breakfast-eaters against breakfast-skippers), look for confounders. If chance decided who got what, the case for cause and effect is much stronger.
- How big is the effect, in real numbers? "Doubles the risk" sounds alarming. But if a rare illness affects 1 person in 10,000, doubling the risk makes it 2 in 10,000: one extra case for every 10,000 people. "Doubles" is a relative risk; "one more in 10,000" is the absolute risk, and it is the one that tells you how much to care.
- Could it be chance? Was the study big enough? Was this one finding picked out of many?
- Is it one study, or many? Science builds up evidence slowly. A single surprising study is often not confirmed when others repeat it.
- Does the headline match the study? Headlines are written to be clicked. The study itself, or the researchers' own summary of it, often says something much more careful.
- Who paid, and who benefits? Funding doesn't make a result wrong, but it is a reason to look closely.
Question 5 matters more than many people realised until recently. In 2015 a large team of researchers repeated 100 published psychology studies as closely as they could. Almost all the original studies had reported clear results, but only about a third of the repeats did. Similar projects in other fields have found similar problems. This has been called the replication crisis, and you will study its causes and cures in Multiple testing and reproducibility. It is not a reason to give up on science. It is science working as it should: checking itself, finding its weak spots, and changing how research is done, for example by asking researchers to publish their plans before collecting data.
Many countries have independent fact-checkers who do this kind of checking for a living, such as Full Fact in the United Kingdom, Africa Check across several African countries and Chequeado in Argentina. Their articles are good practice in reading claims carefully.
Optional Christian reflection · truthful numbers
A reflection from the Christian tradition that inspires Lumenary Academy. It is not part of the mathematics and you are welcome to skip it.
The book of Proverbs says that dishonest scales are hateful to God, while accurate weights are his delight (Proverbs 11:1). Weights and measures were how people in the ancient world traded fairly, and a merchant who quietly shaved his weights was stealing from people who trusted him. Statistics are today's weights and measures: people decide what to eat, what to fear and how to vote by numbers they cannot check themselves. Presenting numbers honestly, with their uncertainty, even when a more dramatic chart would win the argument, is a way of respecting the people who will rely on them. You don't need to share this belief to find the craft of honest statistics worth learning.
Common misconceptions
Misconception
"If a number is true, the statement is honest." True numbers can mislead through a stretched axis, a cherry-picked range, a missing base rate or an unfair comparison. Honesty is about the impression left, not just the digits.
Misconception
"Correlation proves nothing." Correlation alone doesn't prove causation, but it is evidence, and strong, repeated, well-understood correlations (like smoking and lung cancer) can add up to a convincing case. The right response to "A is linked to B" is "how else could that link arise?", not "so what?".
Misconception
"A test that is 90% accurate gives the right answer for 90% of the people who test positive." That depends on the base rate. For a rare condition, most positive results can be false alarms even with a good test, as the 10,000 people showed.
Misconception
"Statistics can be made to say anything, so it's pointless." Every trick in this lesson can be caught by asking the right question. Knowing the tricks is what lets you trust the statistics that deserve it.
Exercises
Be a detective: find the trap, then work out what the numbers really say.
1. How statistics can mislead
Warm-upHumanAn advertisement shows a bar chart of customer ratings: Café Kalimera has 4.3 stars and its rival has 4.1 stars. The vertical axis starts at 4.0, so Kalimera's bar looks three times as tall as its rival's. What is misleading?
2. How statistics can mislead
CoreHumanA report notices that cities with more libraries also have more crime. What is the most likely explanation?
3. How statistics can mislead
CoreHuman2,000 people are tested for a condition that 5% of them have. The test correctly says positive for 80% of the people who have the condition, and wrongly says positive for 10% of the people who don't. How many of the 2,000 people test positive in total?
4. How statistics can mislead
CoreHumanIn the previous exercise, 270 of the 2,000 people tested positive, and 80 of those 270 really have the condition. Tariq tests positive. About how likely is it that he has the condition?
5. How statistics can mislead
CoreHumanA headline says: "Late-night screens raise the risk of a rare eye condition by 50%!" The study found the condition in 2 of every 10,000 people who don't use screens late at night, and 3 of every 10,000 who do. How many extra cases is that for every 10,000 people who use screens late at night?
6. How statistics can mislead
CoreHumanA running club has two groups. The 20 members of the beginners' group ran a mean of 50 km last month. The 5 members of the advanced group ran a mean of 100 km. What is the mean distance run by all 25 members?
7. How statistics can mislead
CoreComputationalWrite a function share_sick(people, base_rate, catches, false_alarm) that uses natural frequencies to find what share of the people who test positive really have the condition:
base_rate: the share of people who have it;catches: the share of those people who test positive;false_alarm: the share of people without it who test positive anyway.
The lesson's example, share_sick(10_000, 0.01, 0.90, 0.05), should give about 0.15.
8. How statistics can mislead
StretchComputationalTwo made-up hospitals treat patients with mild and serious illnesses. The numbers are (survived, treated). Replace each ... so that:
overall_riversandoverall_hillsare each hospital's overall survival rate (all survivors ÷ all patients);hills_better_in_eachisTrueif Hills has the higher survival rate for mild patients and for serious patients;hills_better_overallisTrueif Hills has the higher overall rate.
Then read the output and explain the paradox.
Further reading and resources
- Harding Center for Risk LiteracyResearch and plain-language "fact boxes" on the benefits and harms of medical tests and treatments, presented with natural frequencies.
- Spurious Correlations · Tyler VigenA collection of real data series that rise and fall together by coincidence. Funny, and a lasting warning.
- Estimating the reproducibility of psychological science · Open Science Collaboration · 2015A large project that repeated 100 published psychology studies.
- Women's Health Initiative · US National Heart, Lung, and Blood InstituteThe large randomised trials whose results overturned what observational studies had suggested about hormone therapy.
- Full FactAn independent fact-checking charity in the United Kingdom.
- Africa CheckAn independent fact-checking organisation working across several African countries.
- ChequeadoAn independent fact-checking organisation in Argentina, in Spanish.
- Sense about ScienceA charity that helps the public ask for, and weigh, the evidence behind claims.