Skip to content
LumenaryAcademy
Engineering 80 min

Designing programs

Abstraction, modules and interfaces — writing code that other people can understand and change.

Code tested · 5AI-draftedAwaiting review by a computer scientist

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Why design?

A program that works is the beginning, not the end. Successful software is changed for years — new features, new regulations, new scale, new people — and most of what it costs is the cost of those changes. Code is also read many more times than it is written: by reviewers, by the next engineer, by you in six months, having forgotten everything.

Design is the set of decisions that determine how hard those future changes and readings will be. Its central enemy is complexity: anything about the structure of a system that makes it hard to understand or modify. Complexity rarely arrives in one bad decision. It accumulates — one global variable, one special case, one function that does "just one more thing" — until every change touches ten files and breaks two things nobody expected. Fred Brooks distinguished the essential complexity of the problem itself (tax law is complicated) from the accidental complexity we add with our solutions (a tax calculation tangled with database code). Good design cannot remove the first; its job is to keep the second small.

This lesson collects the ideas that working engineers use to do that: abstraction and interfaces, coupling and cohesion, explicit data, composition, a clean separation of logic from effects, deliberate error handling, good names, safe refactoring, reading code you did not write, and the design reviews through which teams make these decisions together.

DefinitionsH

Abstraction and interfaces

Definition— Abstraction and interface

An abstraction is a simplified view of something that omits the details its users do not need. Its interface is everything a user must know to use it correctly: names and types, what it requires (preconditions), what it promises (postconditions), the errors it can raise, and, where it matters, its cost and its behaviour under concurrency. The implementation is everything else.

You use excellent abstractions every day. A Python dict offers a handful of operations — store, look up, delete, iterate — and hides open addressing, resizing, hash perturbation and memory layout. You can use it fluently without knowing any of that (although Hash tables showed you), and CPython's developers have changed the implementation substantially several times without breaking your code.

In 1972 David Parnas compared two ways of splitting a program into modules: by the steps of processing (read input, then transform, then output), or by information hiding — giving each module one design decision that is likely to change, and hiding that decision behind an interface. His point was that the second makes changes local. If the input format changes, only the module that knows the input format changes. Every later idea in this lesson is a refinement of that one.

John Ousterhout adds a useful measure: the best modules are deep — a small, simple interface over a large amount of functionality, like a file system's open, read, write and close. Shallow modules, whose interfaces are nearly as complicated as their implementations (a class that wraps a dictionary and adds a getter and setter for each key), add names to learn without hiding anything.

Note

An interface is more than what you wrote down. With enough users, every observable behaviour — the order of results, the exact wording of an error, the time a call takes — will be relied on by someone, whatever the documentation promises. This observation is known as Hyrum's Law. It is a reason to keep interfaces small, to avoid exposing incidental behaviour, and to change public interfaces only with care.

ExplanationH

Modules: coupling and cohesion

Two classic measures judge how a system is divided into modules — functions, classes, files, packages or services:

  • Coupling is how much modules depend on each other. Low coupling means you can understand, test and change one module without the others.
  • Cohesion is how strongly the contents of one module belong together. High cohesion means a module does one coherent job, and everything for that job is in one place.

The goal is low coupling, high cohesion. Some kinds of coupling are much worse than others:

Coupling, from benign to harmfulExample
Through simple parametersarea(width, height)
Through a shared data structurepassing a whole order object to a function that needs only its total
Through control flagssave(record, also_email=True, dry_run=False), where flags change what the function is
Through global mutable statetwo modules reading and writing a module-level CONFIG dictionary
Through internalscode that reaches into another class's private attributes

A few rules of thumb follow:

  • Depend on interfaces, not implementations. Code that calls storage.save(report) survives a move from local files to cloud storage; code that builds file paths in twelve places does not.
  • Keep dependencies pointing one way. Business logic should not import the web framework; the web layer imports the logic. Cycles between modules mean they cannot be understood, tested or released separately.
  • Pass in what you need. A function that receives its database connection, clock or HTTP client as a parameter can be tested with a fake and reused in another context; one that reaches for globals cannot. This is all dependency injection means.
  • Things that change together belong together. If every new payment method requires edits to five modules, the payment-method concept is scattered: low cohesion.
DefinitionsH

Data classes: making data explicit

Much accidental complexity comes from data whose shape lives only in programmers' heads: a list where d[0] is the minimum and d[3] the count, a dictionary whose keys differ depending on where it was created. Python's data classes make the shape explicit with almost no code:

python
from dataclasses import dataclass

@dataclass(frozen=True)
class Reading:
    station: str
    celsius: float

    def __post_init__(self):
        if not -90.0 <= self.celsius <= 60.0:
            raise ValueError(f"implausible temperature {self.celsius} at {self.station}")

The decorator generates __init__, a readable __repr__ and __eq__. frozen=True makes instances immutable (and hashable), so a Reading cannot change after it has been validated, and can be shared freely. __post_init__ enforces an invariant at construction, so every Reading in the program is known to be plausible — validation happens once, at the boundary, instead of everywhere. A value object like this — defined entirely by its fields, immutable, validated — is one of the most useful patterns in everyday code: Money(amount_cents, currency), EmailAddress, DateRange. Types and functional programming takes the idea further, to making illegal states impossible to represent at all.

ExplanationH

Objects, and composition over inheritance

An object bundles state with the behaviour that maintains its invariants. The point is not "everything is a class" but encapsulation: callers use methods that keep the state valid, and cannot put it into an invalid state by poking at fields. A BankAccount whose balance can be changed only through deposit and withdraw can enforce "never negative" in one place.

Object-oriented languages offer two ways to reuse behaviour: inheritance ("a SavingsAccount is a BankAccount") and composition ("a Notifier has a Channel"). Experience over decades has favoured composition, for concrete reasons:

  • Inheritance couples a subclass to its parent's implementation, not just its interface. A subclass depends on which of the parent's methods call which others — internal details the parent's authors feel free to change. This is the fragile base class problem.
  • Inheritance hierarchies multiply. Notifications by email or SMS, plain or translated, immediate or batched: as subclasses that is 2×2×2=82 \times 2 \times 2 = 8 classes, and one more option doubles it. As composed parts it is 2+2+2=62 + 2 + 2 = 6 small pieces, combined freely.
  • A subclass must honour the parent's promises. Barbara Liskov's substitution principle says code written for the parent must work with any subclass. A Square that inherits from a mutable Rectangle breaks it: setting a rectangle's width must not change its height.

Python's built-in dict shows the fragile base class problem vividly. Its C implementation does not route update, setdefault or the constructor through __setitem__, so a subclass that overrides only __setitem__ is silently bypassed:

python
class LowercaseKeys(dict):
    """Meant to store every key in lower case. Inherits from dict -- and that is the problem."""
    def __setitem__(self, key, value):
        super().__setitem__(key.lower(), value)

cities = LowercaseKeys({"Pune": 1})
cities["Lagos"] = 2
cities.update({"Lima": 3})
cities.setdefault("Osaka", 4)
print("subclass of dict:", cities)          # only 'lagos' was lowered

class CaseInsensitiveRegistry:
    """Composition: owns a dict, exposes only what callers need, enforces its rule everywhere."""
    def __init__(self, items=()):
        self._data = {}
        for key, value in dict(items).items():
            self.add(key, value)

    def add(self, key, value):
        self._data[key.lower()] = value

    def get(self, key, default=None):
        return self._data.get(key.lower(), default)

    def __contains__(self, key):
        return key.lower() in self._data

    def __len__(self):
        return len(self._data)

registry = CaseInsensitiveRegistry({"Pune": 1, "Lagos": 2})
registry.add("LIMA", 3)
print("composed:        ", registry._data)
print("'lima' in registry:", "lima" in registry, "  get('OSAKA', 0):", registry.get("OSAKA", 0))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The composed version has a smaller interface (no pop, update, |= or fromkeys to keep consistent), so its invariant holds by construction. When you do want a dictionary-like class, the standard library provides collections.UserDict, which is designed for subclassing: all its methods go through __setitem__. Inheritance remains the right tool for a genuine "is a" relationship with a stable, documented extension point — an exception hierarchy, or a framework's base class meant to be subclassed. Elsewhere, reach for composition first, and in Python, for protocols: any object with the right methods will do, no common base class required.

IntuitionH

Functional core, imperative shell

The most effective structural habit for everyday programs is to separate deciding from doing:

  • The functional core holds the logic: parsing, validation, calculations, decisions. It consists of pure functions — their output depends only on their inputs, and they change nothing outside themselves. Pure code is easy to test (call it with values, check the result), easy to reason about, and safe to reuse.
  • The imperative shell is a thin outer layer that performs effects: reads files and requests, consults the clock, writes to databases, sends emails. It gathers inputs, calls the core, and acts on the result. It contains as little logic as possible, so it needs only a few integration tests.

Gary Bernhardt named the pattern "functional core, imperative shell"; it is a practical form of the hexagonal ("ports and adapters") architecture used in larger systems. A sign you need it: a function that both computes a late fee and calls date.today(). Pass today in, and the calculation becomes a pure function that can be tested for 29 February without waiting four years.

ExplanationH

Handling errors deliberately

Errors are part of every interface, and most production incidents start in error paths that nobody exercised. Principles that hold across languages:

  • Validate at the boundaries, trust inside. Check external input — requests, files, user data — once, where it enters, and convert it into validated objects (like Reading). Inner code can then rely on invariants instead of re-checking everywhere.
  • Fail fast and loudly on bugs. If an invariant is violated, stop with a clear error near the cause, rather than carrying corrupted data onwards to fail mysteriously later.
  • Never swallow exceptions silently. except Exception: pass turns a crash you could debug into wrong results you cannot. Catch only what you can handle, as narrowly as possible (except FileNotFoundError), and handle it: retry, use a default, or report.
  • Say what happened and where. "Invalid value" helps nobody; "temperature 999.0 at station Nairobi on line 7 is outside −90..60 °C" fixes itself.
  • Raise specific exceptions, and document them. Callers can only handle errors they can distinguish. A small hierarchy (class IngestError(Exception), with subclasses for each case) beats a generic Exception with a message to parse.
  • Release resources on every path. Use with blocks for files, locks and connections, so cleanup happens whether the code returns or raises.

Python uses exceptions for errors that callers might handle. Many other languages — Rust, Go, and the functional languages — return errors as ordinary values that the type system forces callers to examine; Types and functional programming shows this Result style. Either way, the design question is the same: which errors can occur here, and who is in a position to do something about them?

ExplanationH

Naming

Names are the interface of your code to the next reader, and the cheapest design tool you have.

  • Reveal intent and units. timeout_s, price_cents, distance_km prevent whole classes of bugs; t, p and d invite them.
  • Match length to scope. A loop index used for two lines can be i. A module-level constant read in thirty files deserves MAX_UPLOAD_BYTES.
  • Name functions for what they return or do: parse_readings, is_overdue, send_receipt. A function that is hard to name usually does more than one thing.
  • Booleans read as predicates: is_active, has_expired, should_retry. Avoid negatives (not is_not_valid).
  • Use one word per concept. If the codebase says customer in one place, client in another and user in a third for the same thing, every reader must wonder whether they differ. Use the words the domain experts use.
Worked examplesH

Refactoring, worked through

Refactoring means changing the structure of code without changing its behaviour, in small steps, each verified by tests. When the code has no tests, the first step is a characterisation test: a test that records what the code currently does, right or wrong, so you notice if a step changes it.

Here is a typical "before": it works, but it mixes parsing, conversion, validation, aggregation, formatting and printing in one function, uses lists as anonymous records, and accumulates its output in a global — so calling it twice prints the first report again. The "after" separates those concerns into small pure functions with explicit data, plus a one-line shell. The block runs both and checks they print exactly the same report.

python
import contextlib
import io
from dataclasses import dataclass

CSV = """station,unit,value
Nairobi,C,21.5
Nairobi,C,24.0
Ulaanbaatar,F,-4.0
Ulaanbaatar,F,10.4
Hobart,C,
Hobart,C,13.2
Nairobi,C,999
"""

# ------------------------------- before -------------------------------
REPORT_LINES = []

def process(text):
    stations = {}
    for line in text.strip().split("\n")[1:]:
        p = line.split(",")
        if len(p) != 3 or p[2].strip() == "":
            continue
        s = p[0].strip()
        t = float(p[2])
        if p[1].strip() == "F":
            t = (t - 32) * 5 / 9
        if t < -90 or t > 60:
            continue
        if s not in stations:
            stations[s] = [t, t, t, 1]
        else:
            d = stations[s]
            d[0] = min(d[0], t)
            d[1] = max(d[1], t)
            d[2] += t
            d[3] += 1
    for s in sorted(stations):
        d = stations[s]
        REPORT_LINES.append(f"{s:12} n={d[3]}  min {d[0]:.1f}  max {d[1]:.1f}  mean {d[2] / d[3]:.1f}")
    print("\n".join(REPORT_LINES))

# ------------------------------- after --------------------------------
PLAUSIBLE_CELSIUS = (-90.0, 60.0)          # roughly the extremes ever recorded on Earth

@dataclass(frozen=True)
class Reading:
    station: str
    celsius: float

@dataclass(frozen=True)
class StationSummary:
    station: str
    low: float
    high: float
    mean: float
    count: int

def to_celsius(value, unit):
    # Behaviour kept from the original: any unit other than F is treated as Celsius.
    # Rejecting unknown units is a behaviour change, so it belongs in a separate commit.
    return (value - 32) * 5 / 9 if unit == "F" else value

def parse_readings(text):
    """CSV text -> plausible readings; rows with a missing value are skipped."""
    readings = []
    for line in text.strip().splitlines()[1:]:
        fields = [field.strip() for field in line.split(",")]
        if len(fields) != 3 or not fields[2]:
            continue
        station, unit, value = fields
        celsius = to_celsius(float(value), unit)
        low, high = PLAUSIBLE_CELSIUS
        if low <= celsius <= high:
            readings.append(Reading(station, celsius))
    return readings

def summarise(readings):
    by_station = {}
    for reading in readings:
        by_station.setdefault(reading.station, []).append(reading.celsius)
    return [StationSummary(station, min(temps), max(temps), sum(temps) / len(temps), len(temps))
            for station, temps in sorted(by_station.items())]

def format_report(summaries):
    return "\n".join(f"{s.station:12} n={s.count}  min {s.low:.1f}  max {s.high:.1f}  mean {s.mean:.1f}"
                     for s in summaries)

def main(text):                          # the imperative shell: the only code with effects
    print(format_report(summarise(parse_readings(text))))

# ----------------------- characterisation test ------------------------
def captured(fn, *args):
    buffer = io.StringIO()
    with contextlib.redirect_stdout(buffer):
        fn(*args)
    return buffer.getvalue()

before, after = captured(process, CSV), captured(main, CSV)
assert before == after, (before, after)
print(after)
print("Same output before and after the refactoring.")

# The global in the old version leaks between calls; the new version cannot.
print("old version, second call repeats lines:", captured(process, CSV).count("Nairobi"), "Nairobi lines")
print("new version, second call:              ", captured(main, CSV).count("Nairobi"), "Nairobi line")

# The pure core can be tested directly, with no I/O at all.
assert summarise([Reading("Quito", 10.0), Reading("Quito", 14.0)]) == [StationSummary("Quito", 10.0, 14.0, 12.0, 2)]
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

In practice you would reach the "after" through a sequence of small, separately tested and committed steps, each a named refactoring from Martin Fowler's catalogue:

  1. Add the characterisation test (the captured comparison).
  2. Extract function: pull the Fahrenheit conversion out as to_celsius; run the test.
  3. Introduce parameter object / replace array with object: turn [t, t, t, 1] into named fields. Here it became a list of temperatures per station and a StationSummary.
  4. Split phase: separate parsing from summarising from formatting, so each phase has one input and one output.
  5. Remove global state: build the report as a return value, and print only in main.
  6. Rename everything that is still cryptic.

Notice what the refactoring deliberately did not do: fix the treatment of unknown units, or report skipped rows. Those are behaviour changes. Mixing them into a refactoring makes both harder to review and impossible to revert separately. Make them next, each with a test that describes the new behaviour.

ExplanationH

Reading other people's code

Most professional work starts with code someone else wrote, often a lot of it. Reading code is a skill, and it rewards strategy over persistence:

  • Start from the outside. Read the README and any design documents, then find the entry points: main, the request handlers, the command-line interface. Know what the system is for before reading how.
  • Read the data model first. The core types, the database schema, the API definitions. Data structures change less than code, and once you know them, most functions explain themselves.
  • Run it, and read the tests. Tests are examples of intended use. Running the code with a debugger or a few print calls and following one real request end to end teaches more than reading files in alphabetical order.
  • Use history. git log -p on a file and git blame on a puzzling line (from Version control with Git) often reveal the commit, the review and the bug report that explain why the code is the way it is.
  • Write things down. Keep notes and draw a rough diagram of the modules and their dependencies. Your confusion is valuable: the places you struggled are where documentation would help the next person.

And respect what you do not yet understand. G. K. Chesterton's parable of the fence advises against removing a fence across a road until you know why it was put there. An odd-looking check in old code may encode a lesson learned in an outage; find out before deleting it, and if it truly is obsolete, delete it with a commit message explaining how you know.

ExplanationH

Design reviews in industry

For changes larger than a few days' work, most engineering organisations write a design document before writing the code, and review it with peers. Fixing a design flaw in a document costs an afternoon; fixing it after launch costs a migration. Typical sections:

  • Context and problem: what is wrong or missing, for whom, and why now.
  • Goals and non-goals: what success means, and what is explicitly out of scope. Non-goals prevent the most arguments.
  • Proposed design: the main components, interfaces, data model and flows, with diagrams.
  • Alternatives considered: the other options and why they lost. This section shows reviewers that the problem was explored, and saves the next team from re-litigating it.
  • Cross-cutting concerns: security and privacy, reliability and failure modes, observability, performance and cost, migration and rollout, and how to roll back.

Reviewers look for unstated assumptions, failure modes, simpler alternatives and interactions with other systems. Smaller decisions are often captured as short architecture decision records: one page each, recording a decision, its context and its consequences, kept in the repository next to the code. Open-source communities run the same process in public: Python's PEPs and Rust's RFCs are design documents whose reviews, objections and final decisions anyone can read, and they are an excellent way to see how experienced engineers argue about design.

Common misconceptionsH

Common misconceptions

Misconception

"More abstraction is always better design." An abstraction added before the second or third real use case usually guesses wrong, and a wrong abstraction is harder to remove than duplicated code. Wait until the pattern is clear; many engineers follow a "rule of three".

Misconception

"Don't repeat yourself means never write similar code twice." The principle is about knowledge: every fact (a tax rate, a validation rule) should have one authoritative home. Two pieces of code that look alike but change for different reasons are not duplication, and merging them couples things that should evolve separately.

Misconception

"Good code needs no comments." Good code needs fewer comments about what it does, because names and structure say that. Comments about why — the constraint, the rejected alternative, the bug this check prevents — are some of the most valuable text in a codebase, because the code cannot express them.

Misconception

"We'll clean it up later." Sometimes true, and sometimes a deliberate, recorded trade-off is right. But design debt compounds: each shortcut makes the next change slower, and "later" rarely gets a calendar slot. Leave each piece of code a little better than you found it, as part of normal work.

Exercises

Exercises

1. Designing programs

Warm-upHuman

A function waits before retrying a failed upload. Which is the best name for its parameter holding that wait, measured in milliseconds?

Choose one

2. Designing programs

CoreComputational

The library's reminder code mixes deciding with doing: it reads the clock itself. Rewrite overdue_notices(loans, today) as a pure function of its arguments. loans is a list of (member, title, due_date) tuples with datetime.date due dates. Return one message per loan that is past due on today, in the form "Wanjiru: 'Things Fall Apart' is 3 days overdue" (with 1 day, singular, when it is one day). Order the messages by days overdue, most first, and then by member name.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

3. Designing programs

CoreComputational

Write a value object Money as a frozen data class with fields amount_cents and currency.

  • Construction raises TypeError unless amount_cents is an int (and not a bool), and ValueError unless currency is three upper-case letters A–Z (such as "KES").
  • a + b returns a new Money; it raises ValueError if the currencies differ.
  • m.split(n) returns a list of n Money values in the same currency whose amounts differ by at most one cent and add up to m, the larger shares first. It raises ValueError if n < 1.

Instances must be immutable, comparable with ==, and usable as dictionary keys.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

4. Designing programs

StretchComputational

A sender is any object with a method send(message). Write two wrappers that are senders themselves, so they compose:

  • Retrying(inner, attempts): calls inner.send(message); if it raises ConnectionError, tries again, up to attempts calls in total, then re-raises the last error. Other exceptions are not retried. It returns whatever inner.send returned.
  • Counting(inner): forwards to inner.send(message) and counts how many times its own send was called, in an attribute calls (starting at 0), whether or not the call succeeded.

The tests check that Retrying(Counting(flaky), 3) and Counting(Retrying(flaky, 3)) count different things.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

5. Designing programs

CoreHuman

A colleague writes class Stack(list): and adds push and peek methods, so that stacks get pop, len and iteration for free. What is the strongest objection?

Choose one

6. Designing programs

CoreHuman

A reporting module exposes export(report, fmt="csv", email_to=None, compress=False, dry_run=False). Inside, a 200-line function branches on every flag. Which redesign best reduces coupling and raises cohesion?

Choose one

7. Designing programs

Warm-upHuman

While cleaning up an ingestion service you find if station_id == "HBA-07": celsius += 0.4 with no comment. What should you do first?

Choose one

Further reading and resources