Skip to content
LumenaryAcademy
Foundations 75 min

Exploratory data analysis

Look before you model — summaries and distributions, outliers and data errors, relationships and segments, and the data that make summary statistics lie.

Code tested · 5AI-draftedAwaiting review by a data scientist

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Look before you trust

A grocery delivery service operating in three towns, Asante Bay, San Isidro and Kőhegy, asks whether its delivery times are getting worse and whether its bicycle riders are slower than its motorbike riders. The analyst's first command is df.describe(), and the output looks reassuring: an average delivery of 25.8 minutes, an average basket of 724, an average trip of 6.5 km. All three numbers are wrong, and nothing on the screen says so. The maximum delivery time is 999 minutes. The average basket is seventeen times the median. Two deliveries apparently travelled 4,321 km.

Exploratory data analysis (EDA) is the habit of looking at data from several angles before trusting any summary or fitting any model: distributions, extremes, missing values, relationships, segments and changes over time. John Tukey, who named it in his 1977 book, compared it to detective work: you are looking for clues, including clues that the data are not what you were told. The point is not to produce charts for a report; the first charts you make are for you.

This lesson works through a synthetic delivery dataset with realistic problems planted in it, then builds datasets on which every standard summary statistic agrees and every picture disagrees. The statistical background (distributions, quantiles, correlation) is in describing data and distributions in Lumenary Statistics; here we put it to work.

IntuitionH

Exploring is a conversation with the data

EDA has no fixed recipe, but experienced analysts ask roughly the same questions in roughly the same order.

  1. What is one row? One delivery, one customer, one day? Is that true for every row?
  2. How many, and over what? Row counts, the time span, the number of distinct values in each key column.
  3. What does each column look like? Type, range, typical values, extremes, missing values, the shape of its distribution.
  4. Is anything impossible? Negative durations, dates in the future, a 4,321 km bicycle ride.
  5. What changed over time? Many errors start on a particular date, when a system or a definition changed.
  6. How do columns relate? Scatter plots, correlations, averages within bins.
  7. Does the pattern hold within groups? Split by town, by rider type, by month, and see whether the story changes.

Analogy, not a formal construction

A good mechanic does not start by taking the engine apart. They walk round the car, listen to it idle, look at the tyres and the dashboard lights, and ask when the noise started. Most problems are found that way, and the rest are found faster because the mechanic now knows where to look.

Each answer suggests the next question, and some send you back to cleaning (the loop in the data science workflow). Keep a running list of what you checked and what you found; it becomes the "data notes" section of your report, and it is the first thing a reviewer will ask for.

DefinitionsH

Robust summaries, outliers and errors

Definition— quantiles and robust spread

The pp-quantile of a variable is a value below which a fraction pp of the observations lie: the median is the 0.5-quantile, and the quartiles Q1Q_1 and Q3Q_3 are the 0.25- and 0.75-quantiles. The interquartile range is IQR=Q3−Q1\text{IQR} = Q_3 - Q_1. The median absolute deviation is MAD=median⁡i∣xi−median⁡(x)∣\text{MAD} = \operatorname{median}_i |x_i - \operatorname{median}(x)|. Software interpolates between data values in slightly different ways, so quartiles from different tools can differ a little on small samples.

Definition— outlier and data error

An outlier is an observation far from the bulk of the data, judged by a stated rule (for example, more than 1.5×IQR1.5 \times \text{IQR} beyond the quartiles, or a robust z-score 0.6745 (x−median)/MAD0.6745\,(x - \text{median})/\text{MAD} above 3.5 in absolute value). A data error is a value that does not correctly record what happened: a typo, a unit mix-up, a placeholder code, a test record. Outliers can be errors, and errors can hide among ordinary values; the two are different questions.

The mean and standard deviation are not robust: one wild value can move them anywhere. The median, quartiles, IQR and MAD are robust: changing a few values, however extremely, moves them only a little. That is why the first look at a column should use robust summaries, and why a big gap between the mean and the median is itself a clue.

Outlier rules are flags for inspection, not deletion rules. An unusually long delivery may be an error, or it may be the most interesting delivery in the dataset (a road closure, a new rider, a customer far outside town). The question for each flagged value is: did this really happen? If not, it is an error: fix it from the source if you can, otherwise exclude it and say so. If it did happen, it belongs in the analysis, and your methods must cope with it.

ComputationC

Summaries, distributions and errors

python
import numpy as np
import pandas as pd

# Synthetic deliveries from a grocery service in three towns, January to June 2026.
rng = np.random.default_rng(42)
n = 3000
town = rng.choice(["Asante Bay", "San Isidro", "Kőhegy"], n, p=[0.4, 0.35, 0.25])
rider = rng.choice(["bicycle", "motorbike"], n)
dist = np.where(rider == "bicycle", rng.gamma(2.0, 1.0, n), rng.gamma(2.0, 2.6, n)).round(2)
speed = np.where(rider == "bicycle", 0.30, 0.45)                       # km per minute
minutes = (8 + dist / speed + rng.gamma(2.0, 3.0, n)).round(0)
day = pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 181, n), unit="D")
basket = rng.lognormal(3.6, 0.5, n).round(2)
df = pd.DataFrame({"day": day, "town": town, "rider": rider, "km": dist,
                   "minutes": minutes, "basket": basket})

# Planted problems, of the kinds real systems produce.
df.loc[rng.choice(n, 15, replace=False), "minutes"] = 0                  # staff test orders
df.loc[rng.choice(n, 8, replace=False), "minutes"] = 999                 # placeholder: never delivered
late = (df["town"] == "San Isidro") & (df["day"] >= "2026-04-01")
df.loc[late, "basket"] = (df.loc[late, "basket"] * 100).round(0)         # new till records cents
df.loc[rng.choice(n, 2, replace=False), "km"] = 4321.0                   # GPS glitch

print(df[["km", "minutes", "basket"]].describe().round(1).to_string())
print()
m = df["minutes"]
print("minutes == 0:", (m == 0).sum(), "| minutes == 999:", (m == 999).sum(),
      "| km > 100:", (df["km"] > 100).sum())
monthly = df.pivot_table(index=df["day"].dt.month, columns="town", values="basket", aggfunc="median")
print("median basket by month and town:")
print(monthly.round(1).to_string())

clean = df[(m > 0) & (m < 999) & (df["km"] < 100)].copy()
clean.loc[(clean["town"] == "San Isidro") & (clean["day"] >= "2026-04-01"), "basket"] /= 100
q1, med, q3 = clean["minutes"].quantile([0.25, 0.5, 0.75])
print(f"\nafter cleaning: n={len(clean)}, mean={clean['minutes'].mean():.1f}, "
      f"median={med:.0f}, IQR={q3 - q1:.0f}, max={clean['minutes'].max():.0f}")
print("over the 1.5 x IQR fence:", (clean["minutes"] > q3 + 1.5 * (q3 - q1)).sum(), "deliveries")

# A text histogram: the habit of looking, without a plotting library.
counts, edges = np.histogram(clean["minutes"], bins=range(0, 91, 6))
for c, lo in zip(counts, edges):
    print(f"{lo:3.0f}-{lo + 6:<3.0f} {'#' * (c // 20)} {c}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Read the output as the analyst would.

  • The describe table. The maximum delivery time (999) and trip length (4,321 km) are impossible. The minimum delivery time of 0 is impossible too. The basket column's mean (724.5) is seventeen times its median (42.4), so something enormous lives in the upper tail.
  • The counts. Fifteen zero-minute deliveries are staff test orders (asking the operations team confirms it); eight 999s are the system's code for "never delivered", which belong in a separate analysis of failed deliveries rather than an average of delivery times; two GPS glitches produced the 4,321 km trips.
  • Errors over time. The median basket in San Isidro jumps from between 34 and 40 to between 3,400 and 3,900 in April. The town did not get a hundred times richer: its new tills started recording cents. Only a breakdown by time and segment reveals this kind of error; the overall distribution just looks like a long tail.
  • After cleaning. The mean delivery time is 23.3 minutes and the median 22. The text histogram shows a right-skewed distribution: most deliveries take 12 to 30 minutes, with a tail out to 81.
  • The outlier rule. The 1.5×IQR1.5 \times \text{IQR} fence flags 72 deliveries. They are not errors: 68 of the 72 are motorbike deliveries, with a median trip of 14.8 km, to the edges of town. Deleting them would make the service look faster than it is.

A note on tools. In a notebook you would draw the histogram and scatter plots with a plotting library; the browser Python on this site does not include one, so the lesson prints text histograms instead. The equivalent matplotlib code is shown below for reference; it is not run here.

python
# Not run here: matplotlib, for use in your own notebook.
import matplotlib.pyplot as plt

fig, (left, right) = plt.subplots(1, 2, figsize=(9, 3.5))
left.hist(clean["minutes"], bins=range(0, 91, 3))
left.set(xlabel="delivery time (minutes)", ylabel="deliveries")
right.scatter(clean["km"], clean["minutes"], s=4, alpha=0.3)
right.set(xlabel="distance (km)", ylabel="delivery time (minutes)")
fig.tight_layout()
Worked examplesH

When the summary statistics lie

In 1973 the statistician Francis Anscombe published four small datasets that had the same means, the same variances, the same correlation and the same fitted regression line, and that looked nothing alike when plotted. His point was that "numerical calculations are exact, but graphs are rough" was exactly backwards as advice: the graphs show what the numbers hide. In 2017 Justin Matejka and George Fitzmaurice turned the idea into an algorithm that can morph any scatter plot, a dinosaur included, into others with the same summary statistics.

We can build our own. Take any point cloud. Standardise xx; split yy into the part a straight line in xx explains and the residual ee that it misses; rescale the residual to standard deviation 1. Then set

y′=yˉ+sy(r zx+1−r2  e),y' = \bar y + s_y\left(r\, z_x + \sqrt{1 - r^2}\; e\right),

where zxz_x is the standardised xx. Because zxz_x and ee each have mean 0 and standard deviation 1 and are uncorrelated (the residual of a least-squares fit is uncorrelated with the predictor), y′y' has mean exactly yˉ\bar y, standard deviation exactly sys_y and correlation exactly rr with xx, while keeping the original cloud's kind of shape. Four very different starting shapes then share every summary.

python
import numpy as np

def match_summaries(x, y, mean_x=10.0, sd_x=3.0, mean_y=20.0, sd_y=4.0, r=0.8):
    """Rescale a point cloud so that its means, SDs and correlation are exactly the targets,
    keeping its shape: y is rebuilt from x and the part of y that x does not explain."""
    zx = (x - x.mean()) / x.std()
    resid = y - np.polyval(np.polyfit(zx, y, 1), zx)        # what a straight line misses
    e = resid / resid.std()                                 # mean 0, SD 1, uncorrelated with x
    return mean_x + sd_x * zx, mean_y + sd_y * (r * zx + np.sqrt(1 - r * r) * e)

rng = np.random.default_rng(5)
t = np.linspace(0, 1, 12)
shapes = {
    "A  a noisy line": (t, t + rng.normal(0, 0.25, 12)),
    "B  a smooth curve": (t, -(t - 0.75) ** 2),
    "C  a line + one outlier": (t, np.where(np.arange(12) == 9, 3.0, t)),
    "D  one point does it all": (np.where(np.arange(12) == 11, 3.0, 0.0) + rng.normal(0, 0.02, 12),
                                 np.r_[rng.normal(0, 1, 11), 4.0]),
}
print(f"{'dataset':26} {'mean x':>7} {'sd x':>5} {'mean y':>7} {'sd y':>5} {'r':>5}  fitted line")
data = {}
for name, (x0, y0) in shapes.items():
    x, y = match_summaries(x0, y0)
    data[name] = (x, y)
    b, a = np.polyfit(x, y, 1)
    print(f"{name:26} {x.mean():7.2f} {x.std():5.2f} {y.mean():7.2f} {y.std():5.2f} "
          f"{np.corrcoef(x, y)[0, 1]:5.2f}  y = {a:.2f} + {b:.3f} x")

# ...and yet the datasets are nothing alike. One probe: drop each point in turn and recompute r.
print(f"\n{'dataset':26} r without one point: lowest .. highest")
for name, (x, y) in data.items():
    rs = [np.corrcoef(np.delete(x, i), np.delete(y, i))[0, 1] for i in range(len(x))]
    print(f"{name:26} {min(rs):5.2f} .. {max(rs):5.2f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
A: a noisy line5101520102030B: a smooth curve5101520102030C: a line + one outlier5101520102030D: one point does it all5101520102030
The four constructed datasets. Each has mean x 10, SD x 3, mean y 20, SD y 4, correlation 0.80 and the same least-squares line (dashed), y = 9.33 + 1.067x. A is roughly what those numbers suggest; B is a curve that a straight line describes badly; C is an exact line plus one point that drags the fit; D is a vertical stack of eleven points whose correlation comes entirely from one point far to the right.

The last table shows one numerical probe that does tell the datasets apart: drop each point in turn and recompute the correlation. For A and B it moves only between 0.74 and 0.88. For C, removing one point makes the correlation 1.00. For D, removing one point collapses it to 0.09: the whole "strong relationship" is a single observation. Diagnostics like this (leverage and influence) are the subject of multiple regression and diagnostics in Lumenary Statistics. But the simplest diagnostic is the one in the figure: plot the data.

Worked examplesH

Relationships and segments

Now the operational question: are bicycle riders slower? The block regenerates the same deliveries (without the planted errors), measures how delivery time depends on distance, and then compares riders, first overall and then within distance bands.

python
import numpy as np
import pandas as pd

# The same synthetic deliveries as before, without the planted errors.
rng = np.random.default_rng(42)
n = 3000
town = rng.choice(["Asante Bay", "San Isidro", "Kőhegy"], n, p=[0.4, 0.35, 0.25])
rider = rng.choice(["bicycle", "motorbike"], n)
dist = np.where(rider == "bicycle", rng.gamma(2.0, 1.0, n), rng.gamma(2.0, 2.6, n)).round(2)
speed = np.where(rider == "bicycle", 0.30, 0.45)                       # km per minute
minutes = (8 + dist / speed + rng.gamma(2.0, 3.0, n)).round(0)
df = pd.DataFrame({"town": town, "rider": rider, "km": dist, "minutes": minutes})

# Relationship: how does delivery time depend on distance?
print("Pearson r:", round(df["km"].corr(df["minutes"]), 2),
      "| Spearman rank r:", round(df["km"].corr(df["minutes"], method="spearman"), 2))

# Segment: one overall comparison, then the same comparison within distance bands.
print("\nmean minutes by rider (all deliveries):")
print(df.groupby("rider")["minutes"].agg(["mean", "size"]).round(1).to_string())
df["band"] = pd.cut(df["km"], [0, 2, 4, 8, 40], labels=["0-2 km", "2-4 km", "4-8 km", "8+ km"])
table = df.pivot_table(index="band", columns="rider", values="minutes",
                       aggfunc=["mean", "size"], observed=True).round(1)
print("\nmean minutes and number of deliveries, by distance band:")
print(table.to_string())
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Delivery time rises with distance (Pearson correlation 0.83; the Spearman rank correlation of 0.77 measures any increasing relationship, not only a straight-line one). Overall, bicycle deliveries take 20.9 minutes on average and motorbike deliveries 25.8, so bicycles look faster. Within every distance band, motorbikes are faster: 16.6 against 18.0 minutes for trips under 2 km, and 39.1 against 44.0 for trips over 8 km. Both statements are true. Bicycles are mostly sent on short trips (917 of their 1,531 deliveries are under 2 km), and short trips are quick whoever rides them. The overall comparison mixes the effect of the vehicle with the effect of distance.

This reversal is Simpson's paradox: a comparison that holds in every subgroup reverses when the subgroups are pooled, because the groups being compared have different mixes of a third variable (here distance) that also affects the outcome. Its best-known real case is a 1975 study of graduate admissions at the University of California, Berkeley: men were admitted at a higher rate overall, but most departments admitted women at a similar or higher rate, because women had applied more often to the departments that admitted few applicants of either sex. Segmenting is how EDA finds it, and the general theory of when to adjust for a third variable, and when adjusting makes things worse, is causal inference.

A caution about segmenting: the more ways you slice the data, the more patterns you find by chance. The band with 5 bicycle deliveries over 8 km rests on five numbers. Exploration generates hypotheses; confirming them needs fresh data or a pre-specified test, or you walk into the problems described in multiple testing and reproducibility. A practical habit is to set aside part of the data before exploring, and to check the most important finding on it once, at the end.

ExplanationH

A data quality checklist

Before any number from a dataset goes into a report, work through a list like this one and write down what you found. Most items take one line of pandas.

  • Provenance. Who collected the data, how, when and why? What population does it cover, and who is missing from it? Do you have permission to use it for this purpose?
  • Grain. What is one row? Is the supposed key really unique (df.duplicated(subset=key).sum())? Are there exact duplicate rows?
  • Completeness. Missing values per column, and whether missingness is related to anything: values missing more often in one town, one month or one kind of customer bias every average.
  • Validity. Types, ranges and allowed values: no negative durations, no future dates, categories from the known list, sentinel codes (0, 999, −1, 1900-01-01) identified.
  • Units and definitions. Currency and unit of every number, time zones, and whether a definition changed during the period ("delivered" meaning handed over, or left at the door?).
  • Consistency over time. Counts and medians by week or month: sudden jumps usually mean a system change rather than a real change.
  • Consistency across sources. Totals reconciled with another system (finance, a published figure), and join row counts checked (wrangling data with pandas).
  • Leakage and timing. For any later model: is each column known at the time the prediction will be made? A "delivered on time" flag is no use for predicting delivery time.
  • People. Does the table contain personal data that the analysis does not need? Drop or aggregate it early; exploration does not require names, addresses or phone numbers (see responsible data science).

In professional teams these checks become code that runs every time the data are refreshed: libraries such as pandera and Great Expectations express them as declarative rules, profiling tools such as ydata-profiling generate a first exploratory report of every column, and data engineers build the same checks into pipelines (data quality and governance). Automated reports are a fine start and a poor substitute for thinking: they cannot know that San Isidro changed its tills in April.

Common misconceptionsH

Common misconceptions

Misconception

"Outliers should be removed." Outlier rules flag values for inspection. Remove a value only when you have reason to believe it is an error, and report what you removed. The 72 flagged deliveries in this lesson were real, and removing them would have understated delivery times.

Misconception

"The same summary statistics mean the same data." Means, standard deviations, correlations and fitted lines can all be identical for a straight line, a curve, a line with one outlier and a single high-leverage point. Plot the data.

Misconception

"If a pattern holds overall, it holds in each group (and the other way round)." Simpson's paradox shows that comparisons can reverse when groups with different mixes of a third variable are pooled. Segment by the variables that plausibly drive the outcome.

Misconception

"EDA is just making charts." It is a disciplined search for errors and structure, guided by questions, and it includes counting, cross-checking and talking to the people who produced the data. Its most valuable output is often a list of problems.

Exercises

Exercises

1. Exploratory data analysis

Warm-upHuman

Eleven delivery times, in minutes, sorted: 12, 14, 15, 17, 18, 20, 21, 23, 25, 28, 64.

Using quartiles computed by linear interpolation (NumPy's and pandas's default: the pp-quantile sits at position p(n−1)p(n-1) in the sorted list, counting from 0), what is the upper fence Q3+1.5×IQRQ_3 + 1.5 \times \text{IQR}?

2. Exploratory data analysis

CoreHuman

Which of these are most likely data errors rather than genuine but unusual observations? Select all that apply.

Choose all that apply

3. Exploratory data analysis

CoreHuman

Two datasets of 50 points each have the same means, the same standard deviations, the same correlation (r=0.8r = 0.8) and the same least-squares line. What can you conclude?

Choose one

4. Exploratory data analysis

CoreHuman

A hospital network reports that Clinic North has a lower overall complication rate than Clinic South. Within each severity group (mild, moderate, severe), Clinic South has the lower rate. Which explanation fits?

Choose one

5. Exploratory data analysis

Warm-upHuman

Which statistician named exploratory data analysis in his 1977 book of that title, and also gave us the box plot? (Surname is enough.)

6. Exploratory data analysis

CoreComputational

Write modified_z(values), which returns a NumPy array of robust z-scores

zi=0.6745 (xi−x~)MAD,MAD=median⁡i∣xi−x~∣,z_i = \frac{0.6745\,(x_i - \tilde x)}{\text{MAD}}, \qquad \text{MAD} = \operatorname{median}_i |x_i - \tilde x|,

where x~\tilde x is the median. (The factor 0.6745 makes the score comparable with an ordinary z-score for normally distributed data.) If the MAD is zero, raise ValueError. The tests flag values with ∣z∣>3.5|z| > 3.5, and include a case where two extreme values would hide each other from an ordinary z-score.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Exploratory data analysis

CoreComputational

Write quality_report(df), which returns a dictionary with four entries:

  • "rows": the number of rows;
  • "duplicate_rows": the number of rows that exactly repeat an earlier row;
  • "missing": a dictionary from column name to number of missing values, only for columns with at least one;
  • "constant_columns": a sorted list of columns with at most one distinct non-missing value (including columns that are entirely missing).

All counts must be plain Python ints.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources