Look before you trust
A grocery delivery service operating in three towns, Asante Bay, San Isidro and Kőhegy, asks whether its delivery times are getting worse and whether its bicycle riders are slower than its motorbike riders. The analyst's first command is df.describe(), and the output looks reassuring: an average delivery of 25.8 minutes, an average basket of 724, an average trip of 6.5 km. All three numbers are wrong, and nothing on the screen says so. The maximum delivery time is 999 minutes. The average basket is seventeen times the median. Two deliveries apparently travelled 4,321 km.
Exploratory data analysis (EDA) is the habit of looking at data from several angles before trusting any summary or fitting any model: distributions, extremes, missing values, relationships, segments and changes over time. John Tukey, who named it in his 1977 book, compared it to detective work: you are looking for clues, including clues that the data are not what you were told. The point is not to produce charts for a report; the first charts you make are for you.
This lesson works through a synthetic delivery dataset with realistic problems planted in it, then builds datasets on which every standard summary statistic agrees and every picture disagrees. The statistical background (distributions, quantiles, correlation) is in describing data and distributions in Lumenary Statistics; here we put it to work.
Exploring is a conversation with the data
EDA has no fixed recipe, but experienced analysts ask roughly the same questions in roughly the same order.
- What is one row? One delivery, one customer, one day? Is that true for every row?
- How many, and over what? Row counts, the time span, the number of distinct values in each key column.
- What does each column look like? Type, range, typical values, extremes, missing values, the shape of its distribution.
- Is anything impossible? Negative durations, dates in the future, a 4,321 km bicycle ride.
- What changed over time? Many errors start on a particular date, when a system or a definition changed.
- How do columns relate? Scatter plots, correlations, averages within bins.
- Does the pattern hold within groups? Split by town, by rider type, by month, and see whether the story changes.
Analogy, not a formal construction
A good mechanic does not start by taking the engine apart. They walk round the car, listen to it idle, look at the tyres and the dashboard lights, and ask when the noise started. Most problems are found that way, and the rest are found faster because the mechanic now knows where to look.
Each answer suggests the next question, and some send you back to cleaning (the loop in the data science workflow). Keep a running list of what you checked and what you found; it becomes the "data notes" section of your report, and it is the first thing a reviewer will ask for.
Robust summaries, outliers and errors
Definition— quantiles and robust spread
The -quantile of a variable is a value below which a fraction of the observations lie: the median is the 0.5-quantile, and the quartiles and are the 0.25- and 0.75-quantiles. The interquartile range is . The median absolute deviation is . Software interpolates between data values in slightly different ways, so quartiles from different tools can differ a little on small samples.
Definition— outlier and data error
An outlier is an observation far from the bulk of the data, judged by a stated rule (for example, more than beyond the quartiles, or a robust z-score above 3.5 in absolute value). A data error is a value that does not correctly record what happened: a typo, a unit mix-up, a placeholder code, a test record. Outliers can be errors, and errors can hide among ordinary values; the two are different questions.
The mean and standard deviation are not robust: one wild value can move them anywhere. The median, quartiles, IQR and MAD are robust: changing a few values, however extremely, moves them only a little. That is why the first look at a column should use robust summaries, and why a big gap between the mean and the median is itself a clue.
Outlier rules are flags for inspection, not deletion rules. An unusually long delivery may be an error, or it may be the most interesting delivery in the dataset (a road closure, a new rider, a customer far outside town). The question for each flagged value is: did this really happen? If not, it is an error: fix it from the source if you can, otherwise exclude it and say so. If it did happen, it belongs in the analysis, and your methods must cope with it.
Summaries, distributions and errors
import numpy as np
import pandas as pd
# Synthetic deliveries from a grocery service in three towns, January to June 2026.
rng = np.random.default_rng(42)
n = 3000
town = rng.choice(["Asante Bay", "San Isidro", "Kőhegy"], n, p=[0.4, 0.35, 0.25])
rider = rng.choice(["bicycle", "motorbike"], n)
dist = np.where(rider == "bicycle", rng.gamma(2.0, 1.0, n), rng.gamma(2.0, 2.6, n)).round(2)
speed = np.where(rider == "bicycle", 0.30, 0.45) # km per minute
minutes = (8 + dist / speed + rng.gamma(2.0, 3.0, n)).round(0)
day = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 181, n), unit="D")
basket = rng.lognormal(3.6, 0.5, n).round(2)
df = pd.DataFrame({"day": day, "town": town, "rider": rider, "km": dist,
"minutes": minutes, "basket": basket})
# Planted problems, of the kinds real systems produce.
df.loc[rng.choice(n, 15, replace=False), "minutes"] = 0 # staff test orders
df.loc[rng.choice(n, 8, replace=False), "minutes"] = 999 # placeholder: never delivered
late = (df["town"] == "San Isidro") & (df["day"] >= "2026-04-01")
df.loc[late, "basket"] = (df.loc[late, "basket"] * 100).round(0) # new till records cents
df.loc[rng.choice(n, 2, replace=False), "km"] = 4321.0 # GPS glitch
print(df[["km", "minutes", "basket"]].describe().round(1).to_string())
print()
m = df["minutes"]
print("minutes == 0:", (m == 0).sum(), "| minutes == 999:", (m == 999).sum(),
"| km > 100:", (df["km"] > 100).sum())
monthly = df.pivot_table(index=df["day"].dt.month, columns="town", values="basket", aggfunc="median")
print("median basket by month and town:")
print(monthly.round(1).to_string())
clean = df[(m > 0) & (m < 999) & (df["km"] < 100)].copy()
clean.loc[(clean["town"] == "San Isidro") & (clean["day"] >= "2026-04-01"), "basket"] /= 100
q1, med, q3 = clean["minutes"].quantile([0.25, 0.5, 0.75])
print(f"\nafter cleaning: n={len(clean)}, mean={clean['minutes'].mean():.1f}, "
f"median={med:.0f}, IQR={q3 - q1:.0f}, max={clean['minutes'].max():.0f}")
print("over the 1.5 x IQR fence:", (clean["minutes"] > q3 + 1.5 * (q3 - q1)).sum(), "deliveries")
# A text histogram: the habit of looking, without a plotting library.
counts, edges = np.histogram(clean["minutes"], bins=range(0, 91, 6))
for c, lo in zip(counts, edges):
print(f"{lo:3.0f}-{lo + 6:<3.0f} {'#' * (c // 20)} {c}")Read the output as the analyst would.
- The
describetable. The maximum delivery time (999) and trip length (4,321 km) are impossible. The minimum delivery time of 0 is impossible too. The basket column's mean (724.5) is seventeen times its median (42.4), so something enormous lives in the upper tail. - The counts. Fifteen zero-minute deliveries are staff test orders (asking the operations team confirms it); eight 999s are the system's code for "never delivered", which belong in a separate analysis of failed deliveries rather than an average of delivery times; two GPS glitches produced the 4,321 km trips.
- Errors over time. The median basket in San Isidro jumps from between 34 and 40 to between 3,400 and 3,900 in April. The town did not get a hundred times richer: its new tills started recording cents. Only a breakdown by time and segment reveals this kind of error; the overall distribution just looks like a long tail.
- After cleaning. The mean delivery time is 23.3 minutes and the median 22. The text histogram shows a right-skewed distribution: most deliveries take 12 to 30 minutes, with a tail out to 81.
- The outlier rule. The fence flags 72 deliveries. They are not errors: 68 of the 72 are motorbike deliveries, with a median trip of 14.8 km, to the edges of town. Deleting them would make the service look faster than it is.
A note on tools. In a notebook you would draw the histogram and scatter plots with a plotting library; the browser Python on this site does not include one, so the lesson prints text histograms instead. The equivalent matplotlib code is shown below for reference; it is not run here.
# Not run here: matplotlib, for use in your own notebook.
import matplotlib.pyplot as plt
fig, (left, right) = plt.subplots(1, 2, figsize=(9, 3.5))
left.hist(clean["minutes"], bins=range(0, 91, 3))
left.set(xlabel="delivery time (minutes)", ylabel="deliveries")
right.scatter(clean["km"], clean["minutes"], s=4, alpha=0.3)
right.set(xlabel="distance (km)", ylabel="delivery time (minutes)")
fig.tight_layout()When the summary statistics lie
In 1973 the statistician Francis Anscombe published four small datasets that had the same means, the same variances, the same correlation and the same fitted regression line, and that looked nothing alike when plotted. His point was that "numerical calculations are exact, but graphs are rough" was exactly backwards as advice: the graphs show what the numbers hide. In 2017 Justin Matejka and George Fitzmaurice turned the idea into an algorithm that can morph any scatter plot, a dinosaur included, into others with the same summary statistics.
We can build our own. Take any point cloud. Standardise ; split into the part a straight line in explains and the residual that it misses; rescale the residual to standard deviation 1. Then set
where is the standardised . Because and each have mean 0 and standard deviation 1 and are uncorrelated (the residual of a least-squares fit is uncorrelated with the predictor), has mean exactly , standard deviation exactly and correlation exactly with , while keeping the original cloud's kind of shape. Four very different starting shapes then share every summary.
import numpy as np
def match_summaries(x, y, mean_x=10.0, sd_x=3.0, mean_y=20.0, sd_y=4.0, r=0.8):
"""Rescale a point cloud so that its means, SDs and correlation are exactly the targets,
keeping its shape: y is rebuilt from x and the part of y that x does not explain."""
zx = (x - x.mean()) / x.std()
resid = y - np.polyval(np.polyfit(zx, y, 1), zx) # what a straight line misses
e = resid / resid.std() # mean 0, SD 1, uncorrelated with x
return mean_x + sd_x * zx, mean_y + sd_y * (r * zx + np.sqrt(1 - r * r) * e)
rng = np.random.default_rng(5)
t = np.linspace(0, 1, 12)
shapes = {
"A a noisy line": (t, t + rng.normal(0, 0.25, 12)),
"B a smooth curve": (t, -(t - 0.75) ** 2),
"C a line + one outlier": (t, np.where(np.arange(12) == 9, 3.0, t)),
"D one point does it all": (np.where(np.arange(12) == 11, 3.0, 0.0) + rng.normal(0, 0.02, 12),
np.r_[rng.normal(0, 1, 11), 4.0]),
}
print(f"{'dataset':26} {'mean x':>7} {'sd x':>5} {'mean y':>7} {'sd y':>5} {'r':>5} fitted line")
data = {}
for name, (x0, y0) in shapes.items():
x, y = match_summaries(x0, y0)
data[name] = (x, y)
b, a = np.polyfit(x, y, 1)
print(f"{name:26} {x.mean():7.2f} {x.std():5.2f} {y.mean():7.2f} {y.std():5.2f} "
f"{np.corrcoef(x, y)[0, 1]:5.2f} y = {a:.2f} + {b:.3f} x")
# ...and yet the datasets are nothing alike. One probe: drop each point in turn and recompute r.
print(f"\n{'dataset':26} r without one point: lowest .. highest")
for name, (x, y) in data.items():
rs = [np.corrcoef(np.delete(x, i), np.delete(y, i))[0, 1] for i in range(len(x))]
print(f"{name:26} {min(rs):5.2f} .. {max(rs):5.2f}")The last table shows one numerical probe that does tell the datasets apart: drop each point in turn and recompute the correlation. For A and B it moves only between 0.74 and 0.88. For C, removing one point makes the correlation 1.00. For D, removing one point collapses it to 0.09: the whole "strong relationship" is a single observation. Diagnostics like this (leverage and influence) are the subject of multiple regression and diagnostics in Lumenary Statistics. But the simplest diagnostic is the one in the figure: plot the data.
Relationships and segments
Now the operational question: are bicycle riders slower? The block regenerates the same deliveries (without the planted errors), measures how delivery time depends on distance, and then compares riders, first overall and then within distance bands.
import numpy as np
import pandas as pd
# The same synthetic deliveries as before, without the planted errors.
rng = np.random.default_rng(42)
n = 3000
town = rng.choice(["Asante Bay", "San Isidro", "Kőhegy"], n, p=[0.4, 0.35, 0.25])
rider = rng.choice(["bicycle", "motorbike"], n)
dist = np.where(rider == "bicycle", rng.gamma(2.0, 1.0, n), rng.gamma(2.0, 2.6, n)).round(2)
speed = np.where(rider == "bicycle", 0.30, 0.45) # km per minute
minutes = (8 + dist / speed + rng.gamma(2.0, 3.0, n)).round(0)
df = pd.DataFrame({"town": town, "rider": rider, "km": dist, "minutes": minutes})
# Relationship: how does delivery time depend on distance?
print("Pearson r:", round(df["km"].corr(df["minutes"]), 2),
"| Spearman rank r:", round(df["km"].corr(df["minutes"], method="spearman"), 2))
# Segment: one overall comparison, then the same comparison within distance bands.
print("\nmean minutes by rider (all deliveries):")
print(df.groupby("rider")["minutes"].agg(["mean", "size"]).round(1).to_string())
df["band"] = pd.cut(df["km"], [0, 2, 4, 8, 40], labels=["0-2 km", "2-4 km", "4-8 km", "8+ km"])
table = df.pivot_table(index="band", columns="rider", values="minutes",
aggfunc=["mean", "size"], observed=True).round(1)
print("\nmean minutes and number of deliveries, by distance band:")
print(table.to_string())Delivery time rises with distance (Pearson correlation 0.83; the Spearman rank correlation of 0.77 measures any increasing relationship, not only a straight-line one). Overall, bicycle deliveries take 20.9 minutes on average and motorbike deliveries 25.8, so bicycles look faster. Within every distance band, motorbikes are faster: 16.6 against 18.0 minutes for trips under 2 km, and 39.1 against 44.0 for trips over 8 km. Both statements are true. Bicycles are mostly sent on short trips (917 of their 1,531 deliveries are under 2 km), and short trips are quick whoever rides them. The overall comparison mixes the effect of the vehicle with the effect of distance.
This reversal is Simpson's paradox: a comparison that holds in every subgroup reverses when the subgroups are pooled, because the groups being compared have different mixes of a third variable (here distance) that also affects the outcome. Its best-known real case is a 1975 study of graduate admissions at the University of California, Berkeley: men were admitted at a higher rate overall, but most departments admitted women at a similar or higher rate, because women had applied more often to the departments that admitted few applicants of either sex. Segmenting is how EDA finds it, and the general theory of when to adjust for a third variable, and when adjusting makes things worse, is causal inference.
A caution about segmenting: the more ways you slice the data, the more patterns you find by chance. The band with 5 bicycle deliveries over 8 km rests on five numbers. Exploration generates hypotheses; confirming them needs fresh data or a pre-specified test, or you walk into the problems described in multiple testing and reproducibility. A practical habit is to set aside part of the data before exploring, and to check the most important finding on it once, at the end.
A data quality checklist
Before any number from a dataset goes into a report, work through a list like this one and write down what you found. Most items take one line of pandas.
- Provenance. Who collected the data, how, when and why? What population does it cover, and who is missing from it? Do you have permission to use it for this purpose?
- Grain. What is one row? Is the supposed key really unique (
df.duplicated(subset=key).sum())? Are there exact duplicate rows? - Completeness. Missing values per column, and whether missingness is related to anything: values missing more often in one town, one month or one kind of customer bias every average.
- Validity. Types, ranges and allowed values: no negative durations, no future dates, categories from the known list, sentinel codes (0, 999, −1, 1900-01-01) identified.
- Units and definitions. Currency and unit of every number, time zones, and whether a definition changed during the period ("delivered" meaning handed over, or left at the door?).
- Consistency over time. Counts and medians by week or month: sudden jumps usually mean a system change rather than a real change.
- Consistency across sources. Totals reconciled with another system (finance, a published figure), and join row counts checked (wrangling data with pandas).
- Leakage and timing. For any later model: is each column known at the time the prediction will be made? A "delivered on time" flag is no use for predicting delivery time.
- People. Does the table contain personal data that the analysis does not need? Drop or aggregate it early; exploration does not require names, addresses or phone numbers (see responsible data science).
In professional teams these checks become code that runs every time the data are refreshed: libraries such as pandera and Great Expectations express them as declarative rules, profiling tools such as ydata-profiling generate a first exploratory report of every column, and data engineers build the same checks into pipelines (data quality and governance). Automated reports are a fine start and a poor substitute for thinking: they cannot know that San Isidro changed its tills in April.
Common misconceptions
Misconception
"Outliers should be removed." Outlier rules flag values for inspection. Remove a value only when you have reason to believe it is an error, and report what you removed. The 72 flagged deliveries in this lesson were real, and removing them would have understated delivery times.
Misconception
"The same summary statistics mean the same data." Means, standard deviations, correlations and fitted lines can all be identical for a straight line, a curve, a line with one outlier and a single high-leverage point. Plot the data.
Misconception
"If a pattern holds overall, it holds in each group (and the other way round)." Simpson's paradox shows that comparisons can reverse when groups with different mixes of a third variable are pooled. Segment by the variables that plausibly drive the outcome.
Misconception
"EDA is just making charts." It is a disciplined search for errors and structure, guided by questions, and it includes counting, cross-checking and talking to the people who produced the data. Its most valuable output is often a list of problems.
Exercises
1. Exploratory data analysis
Warm-upHumanEleven delivery times, in minutes, sorted: 12, 14, 15, 17, 18, 20, 21, 23, 25, 28, 64.
Using quartiles computed by linear interpolation (NumPy's and pandas's default: the -quantile sits at position in the sorted list, counting from 0), what is the upper fence ?
2. Exploratory data analysis
CoreHumanWhich of these are most likely data errors rather than genuine but unusual observations? Select all that apply.
3. Exploratory data analysis
CoreHumanTwo datasets of 50 points each have the same means, the same standard deviations, the same correlation () and the same least-squares line. What can you conclude?
4. Exploratory data analysis
CoreHumanA hospital network reports that Clinic North has a lower overall complication rate than Clinic South. Within each severity group (mild, moderate, severe), Clinic South has the lower rate. Which explanation fits?
5. Exploratory data analysis
Warm-upHumanWhich statistician named exploratory data analysis in his 1977 book of that title, and also gave us the box plot? (Surname is enough.)
6. Exploratory data analysis
CoreComputationalWrite modified_z(values), which returns a NumPy array of robust z-scores
where is the median. (The factor 0.6745 makes the score comparable with an ordinary z-score for normally distributed data.) If the MAD is zero, raise ValueError. The tests flag values with , and include a case where two extreme values would hide each other from an ordinary z-score.
7. Exploratory data analysis
CoreComputationalWrite quality_report(df), which returns a dictionary with four entries:
"rows": the number of rows;"duplicate_rows": the number of rows that exactly repeat an earlier row;"missing": a dictionary from column name to number of missing values, only for columns with at least one;"constant_columns": a sorted list of columns with at most one distinct non-missing value (including columns that are entirely missing).
All counts must be plain Python ints.
Further reading and resources
- Exploratory Data Analysis · John W. Tukey · 1977The book that named the field and argued that looking at data deserves as much care as confirming hypotheses.
- Graphs in statistical analysis · F. J. Anscombe · 1973The original four datasets with identical summaries and very different shapes.
- Same stats, different graphs: generating datasets with varied appearance and identical statistics through simulated annealing · Justin Matejka and George Fitzmaurice · 2017The "Datasaurus Dozen", which turns Anscombe's idea into an algorithm.
- Sex bias in graduate admissions: data from Berkeley · P. J. Bickel, E. A. Hammel and J. W. O'Connell · 1975The best-known real case of Simpson's paradox.
- NIST/SEMATECH e-Handbook of Statistical Methods: exploratory data analysis
- panderaDeclarative checks on DataFrames (types, ranges, uniqueness) that turn a quality checklist into code.
- ydata-profilingGenerates an exploratory report of every column in a DataFrame.