A question arrives on a Monday
Ama runs operations for a cooperative of four bakeries: Hilltop, Station, Market and Riverside. Her message to the cooperative's new data scientist is one line long: "Can you look at the sales data? I think we're wasting too much bread."
That message is not yet a question data can answer. "Look at the data" has no end point. "Too much" compared with what? And what would Ama do differently depending on the answer? A good data scientist's first move is not to open a notebook but to ask a few questions back. After a ten-minute conversation, the real situation emerges. Every shop bakes 120 loaves of the standard bread every morning, a habit nobody remembers choosing. A loaf costs 0.40 to make and sells for 1.00 (in the cooperative's currency). Unsold loaves go to a food bank the same evening, which is good, but earns nothing. And Ama has a decision to make before the next baking schedule is printed: how many loaves should each shop bake on each day of the week?
Now there is something to work on. This lesson follows that project from the question to the decision, because the shape of the work (and the ways it goes wrong) is the same whether the data are bread sales, hospital admissions, river levels or clicks on a website. Along the way it sets up the habits the rest of this course relies on: framing the question first, distrusting raw data, stating assumptions, and making every result reproducible.
Data science here means the practice of turning data into decisions. It overlaps with statistics (the methods for reasoning under uncertainty, taught in Lumenary Statistics), with software engineering (the code has to be right and has to keep running), and with the subject matter of whoever asked the question. None of these alone is enough: a perfect model of the wrong question helps nobody, and a well-framed question answered with buggy code is worse than no answer at all.
The workflow, stage by stage
Most descriptions of the work agree on roughly the same stages. Here they are with what each one is for.
- Question. Agree what decision the analysis serves, who makes it, and what would change their mind. Write it down in one or two sentences.
- Data. Find out what data exist, how they were recorded, by whom and why, and whether you are allowed to use them. Often the data you need do not exist yet.
- Clean. Fix or remove what is wrong (duplicates, typos, inconsistent names, impossible values) and record every change.
- Explore. Look at distributions, relationships and oddities before trusting any summary. This is where most surprises appear.
- Model. Build the simplest thing that answers the question: often a well-chosen average or quantile, sometimes a regression or a machine-learning model.
- Communicate. Explain the answer, its uncertainty and its assumptions to the people who decide, in their language.
- Decide. Act, and plan how you will know whether the decision worked.
The stages have been written down many times. The CRISP-DM process model from the late 1990s (business understanding, data understanding, data preparation, modelling, evaluation, deployment) is still the version taught in many companies; R for Data Science uses import, tidy, transform, visualise, model and communicate. The labels matter less than two facts every version agrees on: the work starts and ends with the people who will use it, and it loops.
Definition— kinds of question
A descriptive question asks what happened ("How many loaves did Market sell on Saturdays in April?"). A predictive question asks what will happen, or what an unobserved value is, assuming the world goes on as before ("How many will Station sell next Saturday?"). A causal question asks what would happen if we changed something ("Would opening an hour earlier increase sales?"). A prescriptive question asks what we should do ("How many loaves should Hilltop bake on Fridays?"), and usually combines a prediction with the costs and values of the people deciding.
The four kinds need different evidence. Descriptive questions need accurate counting. Predictive questions need data from a world like the one you will predict in, and a fair test on data the model has not seen. Causal questions are the hardest: patterns in observational data can be produced by confounding, and the gold standard is an experiment (see causal inference in Lumenary Statistics, and experiments in products later in this course). Many failed analyses answered a predictive question when the decision needed a causal one.
A mini-project, end to end
Example— Question: from 'too much waste' to a baking rule
The cooperative wants, for each shop and day of the week, the number of loaves that maximises expected profit, with the current habit (120 a day) as the baseline to beat. This is the classic newsvendor problem from operations research. Bake one more loaf and it sells with some probability , where is the day's demand and the number baked. The extra loaf earns on average and always costs , so it is worth baking while . The best is therefore the smallest value with : the 60th percentile of demand. In general the target percentile is the critical ratio (price − cost) / price.
Notice what that implies before any data arrive: a profit-maximising plan expects to sell out on about 4 days in 10. If the cooperative also cares about regular customers finding bread (a cost not in our formula), the critical ratio should be higher. That is a judgement for Ama, not for the data scientist, and it should be put to her explicitly.
Data. The tills export one row per shop per day: the date, the shop, the number baked, the number sold, and the time of the last sale. The cooperative is happy for the analysis to use them; they contain no information about customers. (When data do describe people, this is where you check consent, purpose and access before anything else; responsible data science treats that properly.) Sixteen weeks of exports are available. The data below are synthetic, generated with a fixed seed, and include the kinds of mess a real export contains.
Clean. The first block builds the export as CSV text, reads it with pandas, and cleans it one step at a time, printing what each step found.
import io
import numpy as np
import pandas as pd
# A synthetic till export from four bakeries: 16 weeks, one kind of loaf.
rng = np.random.default_rng(2026)
dates = pd.date_range("2026-03-02", periods=16 * 7, freq="D") # starts on a Monday
base = {"Hilltop": 95, "Station": 120, "Market": 105, "Riverside": 70}
weekday = np.array([0.9, 0.9, 0.95, 1.0, 1.15, 1.45, 0.75]) # Mon .. Sun
rows = []
for shop, b in base.items():
for d in dates:
demand = rng.poisson(b * weekday[d.dayofweek])
baked = 120 # the habit: 120 every day
sold = min(demand, baked)
if sold < baked: # open 07:00-19:00
minutes = 12 * 60 - rng.integers(0, 40)
else: # sold out early
minutes = int(12 * 60 * baked / demand)
rows.append([d.strftime("%d/%m/%Y"), shop, baked, sold, f"{7 + minutes // 60:02d}:{minutes % 60:02d}"])
# The mess real exports contain.
rows[17][1] = "hilltop " # inconsistent shop names
rows[230][1] = " MARKET"
rows.append(list(rows[40])) # a day exported twice
rows.append(list(rows[41]))
rows[300][3] = -4 # a refund typed into the sales column
rows[350][3] *= 10 # a slip of the finger: an extra zero
del rows[200] # the till was offline one day
text = "date,shop,baked,sold,last_sale\n" + "\n".join(",".join(map(str, r)) for r in rows)
raw = pd.read_csv(io.StringIO(text))
print("raw rows:", len(raw))
print(raw.head(3).to_string(index=False))
print("shop names as typed:", sorted(raw["shop"].unique()))
# Cleaning, step by step, counting what each step changes.
df = raw.copy()
df["shop"] = df["shop"].str.strip().str.title()
print("shop names after tidying:", sorted(df["shop"].unique()))
dups = df.duplicated().sum()
df = df.drop_duplicates()
bad = ~df["sold"].between(0, df["baked"])
print("exact duplicates dropped:", dups, "| impossible sales:", df.loc[bad, "sold"].tolist())
df = df[~bad].copy()
df["date"] = pd.to_datetime(df["date"], format="%d/%m/%Y")
full = pd.MultiIndex.from_product([sorted(df["shop"].unique()), dates], names=["shop", "date"])
missing = full.difference(df.set_index(["shop", "date"]).index)
print("shop-days missing:", len(missing), [f"{s} {d:%a %d %b}" for s, d in missing])
df["sold_out"] = df["sold"] == df["baked"]
print("rows kept:", len(df), "| share of days that sold out:", round(df["sold_out"].mean(), 2))
print(df.groupby("shop")["sold_out"].mean().round(2).to_string())Each step deserves a comment, because each is a judgement call.
- Six spellings of four shop names became four once spaces were stripped and capitalisation fixed. Tidy names before looking for duplicates, or the duplicate rows hide behind different spellings.
- Two rows were exact duplicates (a day exported twice). Dropping exact duplicates is safe here because one shop cannot have two different rows for the same day; in a table of individual sales, two identical rows might be two real customers buying the same thing.
- Two sales figures were impossible: a refund of 4 typed into the sales column, and 620 sold from 120 baked. We dropped them. We could instead have guessed that 620 was 62, but a guess presented as data is how errors become facts. The better fix is to ask the shop, and meanwhile to record the decision.
- Dropping those rows created gaps. Together with the day the till was offline, three shop-days are now missing, and the code found them by comparing against the full calendar, not by eyeballing.
- Finally, the most important line for the analysis:
sold_out. When a shop sells out, its sales record how much bread it had, not how much people wanted. Station sold out on 45% of days.
Explore. Statisticians call this kind of data censored: on sold-out days we know only that demand was at least 120. Any analysis that treats sales as demand will underestimate demand exactly where it is highest, recommend baking less, cause more sell-outs, and so produce even more censored data. The data can quietly confirm a bad habit.
There is a clue in the export, though: the time of the last sale. If Station sells its 120th loaf at 15:37, about 8.6 hours into a 12-hour day, then (if customers arrive at a steady rate) demand for the whole day was about . That is a modelling assumption, and it should be written down as one.
Model. The second block regenerates the same synthetic world (already clean) and compares three baking plans. It fits each plan on the first 12 weeks and scores it on the last 4. Because the data are synthetic, we know the true demand and can score the plans honestly, something you can never do with real data.
import numpy as np
import pandas as pd
# The same synthetic world as above, regenerated without the mess; here we also keep the
# true demand, which a real bakery never sees.
rng = np.random.default_rng(2026)
dates = pd.date_range("2026-03-02", periods=16 * 7, freq="D")
base = {"Hilltop": 95, "Station": 120, "Market": 105, "Riverside": 70}
weekday = np.array([0.9, 0.9, 0.95, 1.0, 1.15, 1.45, 0.75])
rows = []
for shop, b in base.items():
for d in dates:
demand = rng.poisson(b * weekday[d.dayofweek])
sold = min(demand, 120)
minutes = 720 - rng.integers(0, 40) if sold < 120 else int(720 * 120 / demand)
rows.append((d, shop, sold, minutes, demand))
df = pd.DataFrame(rows, columns=["date", "shop", "sold", "minutes_open", "true_demand"])
df["dow"] = df["date"].dt.dayofweek
df["sold_out"] = df["sold"] == 120
# Model: estimate demand on sold-out days by assuming customers arrive at a steady rate.
df["est_demand"] = np.where(df["sold_out"], df["sold"] * 720 / df["minutes_open"], df["sold"])
train, test = df[df["date"] < "2026-05-25"], df[df["date"] >= "2026-05-25"] # 12 weeks, then 4
PRICE, COST = 1.00, 0.40
ratio = (PRICE - COST) / PRICE # bake the 60th percentile of demand
plans = {
"habit: 120 a day": pd.Series(120.0, index=train.groupby(["shop", "dow"]).size().index),
"60th pct of sales": train.groupby(["shop", "dow"])["sold"].quantile(ratio),
"60th pct of est. demand": train.groupby(["shop", "dow"])["est_demand"].quantile(ratio),
}
def score(plan):
q = np.ceil(plan.reindex(pd.MultiIndex.from_frame(test[["shop", "dow"]])).to_numpy())
d = test["true_demand"].to_numpy()
sold = np.minimum(d, q)
return {"profit": PRICE * sold.sum() - COST * q.sum(), "loaves wasted": (q - sold).sum(),
"sell-out days": int((d > q).sum()), "unmet demand": (d - sold).sum()}
print(f"Test: 4 weeks x 4 shops = {len(test)} shop-days, scored against the true demand")
print(pd.DataFrame({name: score(p) for name, p in plans.items()}).T.round(0).astype(int).to_string())
print()
sat = train[train["dow"] == 5].groupby("shop")[["sold", "est_demand", "true_demand"]].mean().round(0)
print("Saturdays in the training weeks (means):")
print(sat.astype(int).to_string())Read the table row by row. The habit of 120 a day wasted 2,859 loaves in four weeks and still turned away 546 customers, because 120 is too many on quiet days and too few on Saturdays. The plan built on raw sales cut waste to 397 loaves, which would look wonderful on a slide titled "waste down 86%", but it sold out on 54 of 112 shop-days and turned away 708 customers: it learned the censored numbers and baked too little exactly when demand was high. The plan built on estimated demand made the most profit (6,324 against 5,205 for the habit), turned away the fewest customers, and wasted 497 loaves. The Saturday table shows why: where sales were stuck at 120, the time-corrected estimates (138, 153 and 172) land within a loaf of the true means.
Communicate. Ama does not need the code. She needs one page: the recommendation (a table of loaves per shop and weekday), what it is expected to do (more profit, fewer customers turned away, somewhat more waste than the cheapest-looking plan), the assumption it rests on, and what we do not know. Writing that page well is a skill of its own, taught in visualisation and storytelling.
Decide. The honest recommendation is not "switch every shop tomorrow". It is a pilot: use the new plan at two shops for four weeks, keep the old one at the other two, and compare, while recording the hour of every sell-out. A pilot turns a prediction from historical data into a measured result, and it is how the analysis earns trust. (A comparison of two shops against two others is a weak experiment; how to design a stronger one is in experiments in products.)
Reproducibility: notebooks, scripts, environments and versions
An analysis is reproducible if someone else (or you, in six months) can rerun it from the raw data and get the same numbers. It sounds obvious; in practice it is the exception. A 2019 study collected over a million Jupyter notebooks from GitHub and tried to rerun them: only about a quarter of those attempted ran from start to finish without an error, and only a few per cent reproduced the results saved in them (Pimentel and colleagues). The usual causes are mundane: missing packages, changed package versions, files that existed only on the author's computer, and cells run in an order nobody wrote down.
Notebooks and scripts. Notebooks (Jupyter, and similar tools) are excellent for exploring: you see each table and chart next to the code that made it. Their danger is hidden state. Cells can run in any order, a deleted cell's variables live on in memory, and the notebook on screen may not be the program that produced the numbers in it. Scripts run top to bottom every time. A common professional pattern uses both: explore in a notebook, then move anything that matters (loading, cleaning, the model, the figures in the report) into functions in plain .py files with tests, and keep notebooks thin. At the very least, before sharing a notebook, restart the kernel and run every cell from the top.
Environments. "It works on my machine" usually means "with my versions of pandas and NumPy". Record the exact versions in a lock file and recreate the environment from it: uv and pip-tools do this for pip-based projects, conda and pixi environments do it for the conda ecosystem, and a container image fixes the operating system as well. Pandas changes behaviour between major versions (copy-on-write became the default in pandas 3.0), so an unpinned analysis can change its answers without a single line of your code changing.
Version control for code. Every analysis belongs in a git repository from its first day (see version control in Lumenary CS). Commit small, meaningful steps; tag the commit that produced each report you send.
Version control for data. Git is not built for large or frequently changing data files. Keep raw data read-only, never edited by hand, and write every change as code. Record which version of the data each result used: a dated snapshot, a table version in a lakehouse, or a tool such as DVC or lakeFS that versions data alongside code. Even with none of these, a fingerprint (a cryptographic hash of the data) lets you prove that two runs used the same input.
Randomness. Every random number in an analysis (simulations, train/test splits, model initialisation) comes from a seeded generator, and the seed is recorded.
import hashlib
import json
import platform
import numpy as np
import pandas as pd
def make_data(seed):
rng = np.random.default_rng(seed)
return pd.DataFrame({"shop": ["Hilltop", "Station", "Market", "Riverside"] * 2,
"sold": rng.poisson(100, 8)})
def fingerprint(df):
"""A short SHA-256 of the table's CSV form: any change to any value changes it."""
return hashlib.sha256(df.to_csv(index=False).encode()).hexdigest()[:16]
a, b = make_data(2026), make_data(2026)
print("same seed, same data:", a.equals(b), fingerprint(a) == fingerprint(b))
print("different seed: ", make_data(2027).equals(a))
record = {
"python": platform.python_version(), "numpy": np.__version__, "pandas": pd.__version__,
"seed": 2026, "rows": len(a), "data_sha256": fingerprint(a),
}
print(json.dumps(record, indent=2))
a.loc[3, "sold"] += 1 # someone "fixes" one number by hand
print("after one edit, fingerprint matches:", fingerprint(a) == record["data_sha256"])The record printed here (versions, seed, row count, data fingerprint) is the kind of thing worth saving next to every report. Your versions may differ from the author's; the point is that the record tells you so. A small project layout that keeps these habits easy looks like this:
bakery-waste/
README.md the question, who asked, how to run everything
pyproject.toml dependencies; uv.lock (or similar) pins exact versions
data/raw/ exports exactly as received: read-only, fingerprinted
data/processed/ written only by code
src/bakery/ load.py, clean.py, model.py: functions with tests
notebooks/ exploration; thin, restarted and rerun before sharing
reports/ what was sent to Ama, with the commit it came fromWhere the time actually goes
A widely repeated claim says data scientists spend 80% of their time cleaning data. Nobody has measured that number carefully, but practitioner surveys and almost every working data scientist agree on the spirit of it: finding, understanding, cleaning and joining data is the largest part of the job, and modelling is a smaller part than courses suggest. In the bakery project, the model was one line (quantile(ratio)); the work was in noticing that sales are not demand.
Much of the remaining time goes to things that rarely appear in tutorials:
- Conversations. Clarifying the question, learning how the data were recorded (the shop manager knows that the till was offline on 29 May; the CSV does not), agreeing what counts as success, and presenting results.
- Checking. Reconciling your totals with someone else's ("finance says Station sold 41,000 loaves; your table says 39,800"), and tracking down the difference. These reconciliations find more bugs than any test suite.
- Getting access. Requesting permissions, signing data-sharing agreements, waiting for a data engineer to expose a table. On teams with good data infrastructure this is quick; elsewhere it can take weeks. The people who build that infrastructure are data engineers (what data engineers do).
- Maintenance. An analysis that becomes a weekly report, or a model that becomes a service, has to keep running when the data change underneath it (data science in production).
Job titles vary between organisations and countries. Roughly: a data analyst answers descriptive questions with SQL, spreadsheets and dashboards; a data scientist adds statistics, experiments and predictive models; a machine-learning engineer builds and operates models in software systems; an analytics engineer or data engineer builds the pipelines and tables everyone else relies on. In small organisations one person does all of it. In every version, the people who do best are those who can explain clearly what the data can and cannot say.
Common misconceptions
Misconception
"The data tell you what to do." Data tell you what happened under the conditions in which they were recorded. Deciding needs a question, costs and values that come from people (like the 0.40 and 0.60 in the bakery), and often an experiment. The bakery's own records were steering it towards baking less.
Misconception
"The model is the deliverable." The deliverable is a better decision. A simple percentile that Ama understands and trusts is worth more than an accurate model nobody acts on, and the most valuable sentence in the project ("sales are not demand on sold-out days") contained no model at all.
Misconception
"It ran in my notebook, so it is reproducible." It ran once, in an order and an environment you did not record. Restart and run all; pin versions; seed randomness; fingerprint inputs.
Misconception
"More data will fix it." Sixteen more weeks of sales at 120 a day would not reveal Saturday demand at Station, because the data are censored at 120 whatever their quantity. More of the wrong data repeats the same bias with more confidence. Sometimes the fix is different data: the time of the last sale, or a pilot that bakes more.
Exercises
1. The data science workflow
Warm-upHumanThe bakery cooperative sends four questions. Which one is a causal question?
2. The data science workflow
CoreHumanOn ten recent Saturdays when it did not sell out, a shop's demand was
88, 132, 97, 120, 141, 105, 126, 113, 150, 101 loaves.
A loaf costs 0.30 to bake and sells for 1.00; unsold loaves earn nothing. Treating these ten days as equally likely, what is the smallest number of loaves that maximises the average profit?
3. The data science workflow
Warm-upHumanOn days when a shop sells out, the sales figure tells you only that demand was at least the number baked. What do statisticians call data like this? (One word is enough.)
4. The data science workflow
CoreHumanIn the lesson's test weeks, the plan built on raw sales wasted the fewest loaves (397, against 497 for the plan built on estimated demand). Why is it still the worse plan?
5. The data science workflow
CoreHumanWhich of these make an analysis more reproducible? Select all that apply.
6. The data science workflow
CoreComputationalWrite clean_sales(raw) for a till export with columns date (text, day first, like 12/03/2026), shop, baked and sold. It must not modify raw, and must return a new DataFrame in which:
- shop names have surrounding spaces removed and are in title case (
" STATION"becomes"Station"); - exact duplicate rows are removed (after step 1, so that duplicates hidden by spelling are caught);
- rows whose
soldis below 0 or abovebakedare removed; dateis parsed into datetimes, day first;- a boolean column
sold_outisTruewhensold == baked; - the index is reset to 0, 1, 2, …
7. The data science workflow
StretchComputationalWrite best_bake(demands, price, cost). demands is a list of past daily demands (positive integers), each treated as equally likely tomorrow. Baking q loaves on a day with demand d earns price * min(d, q) - cost * q. Return the integer q between 0 and max(demands) that maximises the average profit over the past demands; if several values tie, return the smallest.
Compute it by trying every q (do not just use a percentile formula), then check that your answers agree with the critical-ratio rule from the lesson.
Further reading and resources
- R for Data Science (2nd edition): the whole game · Hadley Wickham, Mine Çetinkaya-Rundel and Garrett GrolemundWritten for R, but its picture of the workflow (import, tidy, transform, visualise, model, communicate) is language-independent.
- Good enough practices in scientific computing · Greg Wilson, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt and Tracy K. Teal · 2017Short, practical and realistic advice on organising data, code and projects.
- Ten simple rules for reproducible computational research · Geir Kjetil Sandve, Anton Nekrutenko, James Taylor and Eivind Hovig · 2013
- A large-scale study about quality and reproducibility of Jupyter notebooks · João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo and Juliana Freire · 2019What happened when the authors tried to rerun notebooks published on GitHub.
- The Turing Way · The Turing Way communityAn open, community-written handbook to reproducible, ethical and collaborative data science.
- Project Jupyter
- uv, a Python package and project manager
- DVC (Data Version Control)