One number is not enough
Akosua runs a bakery in Accra. On an average morning she sells 30 loaves of tea bread. How many should she bake?
Thirty is the obvious answer, and a poor one. Demand varies from morning to morning: some days 24 people want a loaf, some days 37. If she bakes exactly the average, she will turn customers away on close to half of all mornings, as we will calculate below. If she bakes 60, she will rarely run out but will throw bread away almost every day. A sensible answer needs more than the average. It needs to know how often demand reaches 35, 40 or 45 loaves: the whole pattern of variation. That pattern is a distribution.
The same need appears everywhere statistics is used. A maternity ward in Manila plans staffing from the number of births per night, which varies. A machine in Stuttgart turns steel shafts whose diameters vary by a few micrometres around the target, and the question is what fraction will fall outside tolerance. A seismologist in Santiago asks how long the wait until the next large earthquake might be. In each case the useful answer is not one number but a description of which values are possible and how likely each one is.
This lesson introduces the language for that description (random variables, mass and density functions, expectation, variance and quantiles) and the four distributions that a working statistician reaches for first. Each comes with a story, a description of the kind of process that produces it, because the story is how you decide whether a distribution fits your problem. And each comes with a check: we will simulate it and compare the simulation with the formulas, a habit you will use throughout Lumenary Statistics.
Numbers that come from chance
In Chance and probability you met probabilities of events: the chance of rolling a six, or of rain tomorrow. Statistics usually cares about numbers produced by chance: how many of 20 seeds sprout, how many calls arrive in a minute, how tall the next person measured will be, how long until the next bus. A random variable is such a number: a quantity whose value is decided by a chance process. We write it with a capital letter, , and its observed value with a small one, .
Random variables come in two kinds.
- Discrete ones take separate values you can list: 0, 1, 2, 3, … seeds; 0, 1, 2, … calls. Each value has a probability, and we can draw them as bars whose heights add up to 1.
- Continuous ones can take any value in an interval: a height of 171.3 cm, or 171.32, or 171.3247. The chance of any one exact value is zero, so instead we describe how probability is spread along the line with a curve, the density. Probability is then the area under the curve over an interval, and the total area is 1.
A good way to picture any distribution is as the histogram you would get if you could repeat the chance process forever. Roll the process ten times and the histogram is ragged; a thousand times and its shape emerges; a million times and it settles onto the bars or the curve. You met histograms in Describing data as the shape of a data set; a distribution is the shape of a process. We cannot repeat Akosua's mornings a million times, but a computer can repeat a model of them, which is why simulation is the statistician's laboratory.
Definitions
Definition— probability mass function
A discrete random variable has a probability mass function (pmf) , with and .
Definition— probability density function
A continuous random variable has a probability density function (pdf) with total area , such that for any interval The value is not a probability: it is probability per unit of , and it may be larger than 1.
Definition— cumulative distribution function
The cumulative distribution function (cdf) of any random variable is . It rises from 0 to 1. Its complement is called the survival function.
Definition— expectation, variance and standard deviation
The expectation (mean) of is for a discrete variable and for a continuous one: the long-run average of many independent repetitions. The variance is , the average squared distance from the mean, and the standard deviation is , in the same units as .
Definition— quantile
For , the -quantile of is the smallest value with . The 0.5-quantile is the median; the 0.95-quantile is the 95th percentile. Software calls the function the quantile function, inverse cdf or ppf (percent point function).
Expectation is the "expected value" of a game from Chance and probability, extended to any random variable. Three rules do most of the everyday work. For constants and , and : shifting moves the centre but not the spread, and rescaling stretches both. For any random variables, ; and if and are independent, . Lumenary Math proves these in Probability theory; here we use them.
Four distributions and their stories
The binomial: counting successes in a fixed number of tries
Story. Run independent trials, each a success with the same probability , and count the successes. Then , with
The mean and variance follow from writing as a sum of independent 0-or-1 variables, each with mean and variance , and using the rules above.
Example— A germination test
Wanjiru, an agronomist in Nakuru, tests a batch of bean seed by planting 20 seeds; the supplier claims 80% germinate. If the claim is right and seeds sprout independently, the count is Binomial(20, 0.8): expect sprouts, with standard deviation . All 20 sprouting has probability , and 13 or fewer has probability about 0.087, so a result of 13 is unusual but not shocking. Hypothesis testing turns that "unusual but not shocking" into a careful argument.
The story carries conditions: a fixed number of trials, independence, and the same for every trial. Seeds from one diseased pod violate independence; mixing two batches with different germination rates violates "same ". Either way the counts will vary more than the binomial predicts.
The Poisson: counting events in a window
Story. Events happen one at a time, independently, at a constant average rate, and we count how many fall in a fixed window of time or space: calls to a helpline per minute, typing errors per page, flaws per metre of cable. Then , where is the average count per window:
The Poisson is the binomial's limit when there are very many opportunities, each very unlikely: split a minute into tiny slices, each holding a call with probability , and let grow. Siméon-Denis Poisson derived it in 1837, and in 1898 Ladislaus Bortkiewicz showed how well it describes rare events by tabulating the number of soldiers killed by horse kicks in Prussian cavalry corps over twenty years: deaths were rare, independent and about equally likely in any corps-year.
The equality of mean and variance is the Poisson's fingerprint, and a practical check: if your counts have a variance far bigger than their mean, the Poisson story does not hold.
The normal: many small effects added together
Story. A quantity is the sum of many small, independent influences, none of which dominates: a measurement error made of dozens of tiny disturbances, or an adult height shaped by many genes and many meals. Then the quantity is approximately normal (also called Gaussian), with density
Why sums of small effects become normal is the central limit theorem, the subject of the next lesson. Every normal distribution is a shifted, stretched copy of the standard normal : if then , the z-score, is standard normal. So one table of the standard normal serves every normal distribution, and every normal puts about 68% of its probability within one standard deviation of the mean, 95% within two, and 99.7% within three.
The exponential: waiting for the next event
Story. Events arrive as in the Poisson story at rate per unit time, and we measure the waiting time until the next one. Then , with
The two stories are two views of one process. If calls arrive at 3 per minute, the count in each minute is Poisson(3) and the gaps between calls are Exponential(3), with mean of a minute.
Theorem— The exponential distribution is memoryless
If , then for all , Having already waited minutes tells you nothing about how much longer you will wait.
Proof
By the definition of conditional probability, and because the event is contained in ,
∎
Memorylessness is exactly right for radioactive decay, and badly wrong for a light bulb that wears out or for a bus that runs to a timetable. It is the precise sense in which the exponential describes events that "just happen" with no build-up.
| Distribution | Story | Values | Mean | Variance |
|---|---|---|---|---|
| Binomial | successes in independent trials | |||
| Poisson | events in a fixed window, constant rate | |||
| Normal | sum of many small independent effects | all real numbers | ||
| Exponential | waiting time to the next event |
Simulate, then check
A formula you have derived is a claim, and a simulation is a cheap experiment to test it. NumPy's random Generator draws from all four distributions. We fix the seed so that the program prints the same numbers every time, which makes the output reproducible and lets you compare your run with this page.
import math
import numpy as np
rng = np.random.default_rng(2026)
N = 200_000
# Four distributions: draw N values from each and compare with the formulas.
cases = [
("Binomial(n=20, p=0.3)", rng.binomial(20, 0.3, N), 20 * 0.3, 20 * 0.3 * 0.7),
("Poisson(lam=4)", rng.poisson(4.0, N), 4.0, 4.0),
("Normal(mu=170, sd=8)", rng.normal(170.0, 8.0, N), 170.0, 8.0**2),
("Exponential(rate=0.5)", rng.exponential(1 / 0.5, N), 1 / 0.5, 1 / 0.5**2),
]
print(f"{'distribution':22} {'mean: sim':>10} {'formula':>8} {'var: sim':>10} {'formula':>8}")
for name, x, mean, var in cases:
print(f"{name:22} {x.mean():10.3f} {mean:8.3f} {x.var():10.3f} {var:8.3f}")
# The probability mass function of Binomial(10, 0.3), simulated and exact, as a text bar chart.
n, p = 10, 0.3
x = rng.binomial(n, p, N)
print("\nk simulated exact")
for k in range(n + 1):
sim = np.mean(x == k)
exact = math.comb(n, k) * p**k * (1 - p) ** (n - k)
print(f"{k:2} {sim:9.4f} {exact:7.4f} " + "#" * round(100 * exact))Every simulated mean and variance sits within a whisker of its formula, and the simulated binomial frequencies match to about two decimal places. The bar chart also shows the shape: Binomial(10, 0.3) piles up around 3 and has a longer tail to the right than to the left.
Quantiles answer planning questions
Back to the bakery. Suppose Akosua's morning demand is Poisson with mean 30: many independent customers, each unlikely to come on any given morning, at a steady average rate. (That is a modelling assumption; checking it against her sales records would be the first job of a real analysis.) To turn customers away on at most one morning in twenty, she needs the smallest number of loaves with : the 95th percentile of demand.
SciPy's distribution objects answer such questions directly. Their method names are worth learning, because the same names appear across the scientific Python ecosystem: pmf or pdf for the height, cdf for , sf (survival function) for , ppf for the quantile, and rvs for random draws.
import numpy as np
from scipy import stats
# A bakery sells, on average, 30 loaves of tea bread each morning. If demand is Poisson(30),
# how many loaves must it bake to sell out on at most 1 morning in 20?
demand = stats.poisson(30)
k95 = demand.ppf(0.95) # ppf = "percent point function" = quantile
print("95th percentile of Poisson(30):", int(k95))
print("P(demand <= 38) =", round(demand.cdf(38), 4), " P(demand <= 39) =", round(demand.cdf(39), 4))
# The same answer by simulating 100,000 mornings.
rng = np.random.default_rng(7)
mornings = rng.poisson(30, 100_000)
print("simulated 95th percentile:", np.quantile(mornings, 0.95))
print("share of mornings that would sell out with 39 loaves:", np.mean(mornings > 39))
# Familiar quantiles of the standard normal, and the median of an exponential wait.
z = stats.norm()
for q in [0.5, 0.84, 0.95, 0.975, 0.995]:
print(f"standard normal: {q:.3f} quantile = {z.ppf(q):6.3f}")
wait = stats.expon(scale=1 / 0.5) # SciPy uses scale = 1/rate
print("exponential (rate 0.5): mean =", wait.mean(), " median =", round(wait.median(), 3))Thirty-nine loaves. With 38 she would fall short on of mornings; with 39, on about 4.6%. The 100,000 simulated mornings agree, and baking only the average of 30 would leave customers disappointed on about 45% of mornings. The extra nine loaves are the price of reliability, and they come straight from the spread of the distribution: a Poisson with mean 30 has standard deviation , and the 95th percentile of a roughly normal quantity sits about 1.645 standard deviations above the mean, .
The table of standard normal quantiles is worth memorising: 1.645 (95th percentile), 1.96 (97.5th) and 2.576 (99.5th) will reappear in every lesson that follows. Notice too that the exponential's median, minutes, is well below its mean of 2: in a right-skewed distribution a few long waits pull the mean up, while the median stays with the typical case.
The normal approximation, and when it fails
In 1733 Abraham de Moivre, a French Huguenot mathematician who made a modest living in London tutoring mathematics and advising gamblers and insurers, noticed that the binomial bars for large trace out a bell-shaped curve. In modern terms: when is large, To approximate for a count, we use for the normal : the continuity correction, which accounts for each bar covering the interval from to .
Before computers, this approximation was how anyone computed binomial probabilities for large . Today SciPy gives exact binomial tails in microseconds, so the approximation matters for a different reason: it explains why so many statistics are roughly normal, and it is the basis of quick mental calculations and of many standard tests. You need to know when to trust it.
import math
from scipy import stats
def approximations(n, p, k):
"""P(X >= k) for X ~ Binomial(n, p): exact, normal approximation, Poisson approximation."""
exact = stats.binom.sf(k - 1, n, p) # sf(k - 1) = P(X > k - 1) = P(X >= k)
mu, sd = n * p, math.sqrt(n * p * (1 - p))
normal = stats.norm.sf((k - 0.5 - mu) / sd) # with the continuity correction
poisson = stats.poisson.sf(k - 1, mu)
return exact, normal, poisson
print(" n p k P(X>=k) exact normal approx Poisson approx")
for n, p, k in [(100, 0.5, 55), (100, 0.5, 65), (100, 0.5, 70), (100, 0.5, 75),
(40, 0.05, 4), (40, 0.05, 6), (40, 0.05, 8), (2000, 0.001, 6)]:
e, nm, po = approximations(n, p, k)
print(f"{n:4} {p:6.3f} {k:4} {e:13.3g} {nm:13.3g} {po:14.3g}")
# Why the normal struggles with Binomial(40, 0.05): it is lopsided and piled up against zero.
n, p = 40, 0.05
mu, sd = n * p, math.sqrt(n * p * (1 - p))
print(f"\nBinomial({n}, {p}): mean {mu}, sd {sd:.3f}")
for k in range(9):
pk = stats.binom.pmf(k, n, p)
print(f"{k} {pk:.4f} " + "#" * round(100 * pk))
print("a normal curve with this mean and sd puts", round(stats.norm.cdf(-0.5, mu, sd), 3),
"of its probability below -0.5, on impossible negative counts")Two things stand out.
- Symmetric and large: excellent near the centre, worse in the far tails. For Binomial(100, 0.5), the normal approximation is essentially exact for , but at it is too large by about 70% in relative terms ( against ). The absolute error is tiny, but if your decision depends on how rare something is (a one-in-a-million event), relative error is what matters.
- Skewed: poor even at moderate distances. Binomial(40, 0.05) has mean 2 and is piled up against zero. The normal curve with the same mean and standard deviation puts 3.5% of its probability on impossible negative counts, and it underestimates by a factor of about 20. The Poisson approximation, Poisson, does far better for rare events, and for Binomial(2000, 0.001) it is almost exact.
The usual rule of thumb asks for and before using the normal approximation (some books say 5). Treat it as a minimum for approximations near the centre, not as a licence for tail probabilities.
Distributions in professional practice
Choose by story, then check against data. A statistician picks a candidate distribution because its story matches how the data were generated, then checks the fit: a histogram against the fitted curve, a comparison of mean and variance for counts, and a quantile–quantile (Q–Q) plot, which plots the sorted data against the quantiles the model predicts, so that a good fit lies on a straight line and heavy tails bend away at the ends. George Box's often-quoted remark that all models are wrong but some are useful is the working attitude: no real process is exactly binomial or normal, and the question is whether the model is good enough for the decision at hand.
Beyond the four. Count data with extra variation are often modelled with the negative binomial. Quantities that are products of many factors, such as incomes or the sizes of particles after crushing, are often closer to lognormal. Some phenomena, such as the sizes of cities, insurance losses or the popularity of web pages, have heavy tails in which extreme values are far more common than any normal model allows; ignoring that has repeatedly led risk models to understate the chance of large losses. The NIST/SEMATECH e-Handbook describes dozens of distributions with the situations they model.
Software conventions. In Python, scipy.stats provides each distribution as an object with the methods used above. In R, the same four operations are four functions sharing a prefix: d for density or mass, p for the cdf, q for quantiles, r for random draws. This R code is shown for comparison and is not run here:
# R, not run here: the bakery and seed examples
qpois(0.95, lambda = 30) # 39
pbinom(13, size = 20, prob = 0.8) # P(13 or fewer sprout), about 0.087
rexp(5, rate = 0.5) # five random waiting timesReproducible randomness. Simulations in research and industry use seeded generators so that results can be reproduced and audited. NumPy's default generator is PCG64; record the seed with the results, and never choose a seed by trying several until the answer looks good, which quietly turns a simulation into a forking path.
Common misconceptions
Misconception
"Most data are normally distributed." Many are not: counts near zero, waiting times, incomes, reaction times and file sizes are all right-skewed. The normal is common for sums and averages of many independent pieces (the next lesson explains why), not for raw data in general. Look at the data before assuming a shape.
Misconception
"The density at a point is the probability of that value." For a continuous variable every single value has probability zero; the density is probability per unit length. A normal distribution with standard deviation 0.1 has density about 3.99 at its centre, which is fine, because only areas are probabilities.
Misconception
"The mean is the most likely value." For Poisson(0.5) the mean is 0.5, a value that can never occur; the most likely count is 0 (probability 0.61). For the exponential, the density is highest at 0 although the mean is . The mean is a balance point, not a forecast of the typical case.
Misconception
"Random events are spread out evenly." Look at the arrivals in the figure above: five calls in the first minute (three of them within a few seconds) and one in the last. Clusters and gaps are what independence looks like. People who see a cluster of rare events, such as several cases of an illness in one village, often conclude that something must have caused it; sometimes something did, but chance alone produces more clusters than intuition expects.
Misconception
"I've waited so long that the next one must be due." For a memoryless process, the expected remaining wait is the same however long you have already waited. The intuition is right only when the process has memory: buses that keep to a timetable, or a machine part that wears out.
Exercises
1. Distributions
Warm-upHumanA maternity ward in Manila records the number of babies born between midnight and 1 a.m. each night. Which distribution is the natural first model for that count?
2. Distributions
Warm-upHumanA quality inspector in Monterrey checks 400 parts, each defective with probability 0.1, independently. What is the standard deviation of the number of defective parts?
3. Distributions
CoreHumanMinibuses pass a stop in Kampala with no timetable, as a Poisson process at an average of 6 per hour, so the wait from any moment is Exponential with rate 6 per hour. Amara has already waited 15 minutes. On average, how many more minutes will she wait?
4. Distributions
CoreHumanA machine in a mill near Hanoi fills bags of rice. The fill weights are well described by a normal distribution with mean 1000 g and standard deviation 5 g. Above what weight, to the nearest gram, are the heaviest 2.5% of bags?
5. Distributions
CoreHumanMeasurement errors of a precision balance are modelled as Normal with mean 0 and standard deviation 0.1 mg. A student evaluates the density at 0 and gets 3.99. She concludes that she must have made a mistake, because probabilities cannot exceed 1. What is the right response?
6. Distributions
StretchHumanA clinic in Lusaka vaccinates 30 children in a session. Each, independently, has a 2% chance of a mild fever afterwards. The number with fever is Binomial(30, 0.02), and the exact probability that no child has a fever is 0.545. Which approximation to that probability is better, and why?
7. Distributions
CoreComputationalWrite loaves_to_bake(mean, risk), which returns the smallest whole number such that risk when demand is Poisson with the given mean. For example, loaves_to_bake(30, 0.05) should be 39, as in the lesson.
Use only the math module (no SciPy). A helper poisson_pmf is provided, but for large means it is better to build the probabilities one from the previous: .
8. Distributions
StretchComputationalIf is uniform on and is a continuous cdf, then has cdf . This inverse-transform method is how many random draws are made.
- Write
exponential_from_uniform(u, rate): given a number or NumPy arrayuof values in , return the exponential quantiles . - Write
mean_extra_wait(rate, s, n, seed): drawnwaiting times with your function fromnp.random.default_rng(seed).random(n), keep those longer thans, and return the mean of their extra wait beyonds(as a float). The memoryless property predicts about1 / rate, whateversis.
Further reading and resources
- NIST/SEMATECH e-Handbook of Statistical Methods · National Institute of Standards and TechnologyA free handbook of engineering statistics; its chapter on exploratory data analysis has a gallery of common distributions with their uses.
- SciPy: statistical functions (scipy.stats)Every distribution in this lesson, with pmf or pdf, cdf, sf, ppf (quantiles) and rvs (random draws).
- NumPy: random GeneratorHow to make seeded, reproducible random draws.
- OpenIntro Statistics · David Diez, Mine Çetinkaya-Rundel and Christopher BarrA free, openly licensed textbook whose chapters on distributions cover the same ground at a gentler pace.