Skip to content
LumenaryAcademy
Frontier 85 min

The modern data stack in 2026

How the pieces of a data platform fit together today — shared in-memory formats, embedded engines, open tables, zero-ETL and federation, semantic layers and the bill — and what is genuinely new since Hadoop.

Code tested · 5AI-draftedAwaiting review by a data engineer

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Many parts, one platform

A university hospital network in Porto Alegre wants one place to answer questions about its operations: waiting times, bed occupancy, supply costs. The data lives in a patient-administration database, a laboratory system, a stream of events from bedside devices, spreadsheets from suppliers and files from the regional health authority. Ten years ago the default answer was a Hadoop cluster in the basement. In 2026 the team faces a menu instead: a cloud warehouse or a lakehouse, an open table format, a catalogue, perhaps several query engines, an embedded engine for analysts' notebooks, "zero-ETL" replication offered by their database vendor, a semantic layer for shared definitions — and a bill that arrives monthly and can surprise.

"The modern data stack" is a phrase that spread around 2020 among vendors and practitioners for one such assembly: a cloud warehouse, managed connectors loading into it, SQL transformations versioned like code, and BI tools on top. It began partly as marketing, and by 2024 many teams were consolidating tools it had multiplied. This lesson uses the phrase loosely, for the platform as it is assembled in 2026, and examines the pieces newest since the earlier lessons: shared in-memory formats, embedded engines, the state of open tables, zero-ETL and federation, semantic layers, cost, and what has really changed since Hadoop.

IntuitionH

The layers

The deepest change since the Hadoop era is that a platform is no longer one product but layers joined by interfaces, several of them open standards. That lets an organisation change one layer — a query engine, say — without rewriting the others, and lets several engines share one copy of the data.

about 20122026ETL tools, log shippersone Hadoop clusterHive, Pig andMapReduce jobsYARN: resourcesHDFS on the cluster'sown disksHive metastoreBI tools over JDBC and ODBCBI · notebooks · applications · AI assistantssemantic layer: metrics and definitionsware-houseSparkTrinoembeddedenginesstreamprocessorsbetween engines and clients: SQL, Arrow, ADBC, Flightcatalogue: tables, schemas, access rules (REST APIs)open table formats: Iceberg · Delta Lake · Hudi · Paimonobject storage (any cloud or on premises)data files, mostly Parquetingestion: CDC, connectors, zero-ETL
A data platform in about 2012 and in 2026. The Hadoop cluster bundled storage, resources and compute on the same machines; a 2026 platform separates them into layers, with open formats and APIs between them, so that several engines can share one copy of the data.

Read the 2026 stack from the bottom. Object storage holds the bytes, cheaply and durably, mostly as Parquet (Data models and file formats). Open table formats turn folders of files into tables with transactions, snapshots and schema evolution, and a catalogue says which tables exist, where their current metadata is and who may read them (Lakehouses and table formats). Several engines read the same tables: a warehouse for dashboards, Spark for large batch jobs (Distributed processing), a stream processor for fresh data, an embedded engine on an analyst's laptop. A semantic layer defines the business's metrics once. The interfaces between layers — Parquet, the table-format specifications, the catalogue's REST protocol, SQL, Arrow — are what make it a platform rather than a product.

ComputationC

Interchange and Apache Arrow

Every layer boundary moves data. When ten tools each have their own in-memory format, connecting every pair needs up to 10×9=9010 \times 9 = 90 converters, each a place for slowness and bugs; if all speak one format, each needs one adapter. That is the case for Apache Arrow, whose columnar memory layout Data models and file formats took apart buffer by buffer. Here we measure what interchange costs. The block moves 100,000 orders between two "tools" four ways: as Python row tuples (what a row-oriented database driver hands you), as JSON lines, as CSV, and as an Arrow IPC stream.

python
import io
import json
import time
import numpy as np
import pyarrow as pa
import pyarrow.csv as pacsv

rng = np.random.default_rng(4)
n = 100_000
countries = np.array(["KE", "NA", "BR", "IN", "PL", "JP", "NG", "MX"])    # NA is Namibia
table = pa.table({
    "order_id": pa.array(np.arange(10**9, 10**9 + n), pa.int64()),
    "country": pa.array(countries[rng.integers(0, 8, n)]),
    "amount": pa.array(np.round(rng.gamma(2.0, 20.0, n), 2)),
    "ordered_at": pa.array(1_767_225_600_000 + np.sort(rng.integers(0, 90 * 86_400_000, n)), pa.timestamp("ms")),
})

def timed(f):
    t0 = time.perf_counter()
    out = f()
    return out, time.perf_counter() - t0

def via_rows():
    """What a row-oriented driver does: one Python tuple per row, then rebuild the columns."""
    rows = list(zip(*(table.column(c).to_pylist() for c in table.column_names)))
    cols = list(zip(*rows))
    return pa.table({c: pa.array(v, table.schema.field(c).type) for c, v in zip(table.column_names, cols)}), None

def via_json():
    text = "\n".join(json.dumps(r, default=str) for r in table.to_pylist()).encode()
    back = [json.loads(line) for line in text.splitlines()]
    return pa.Table.from_pylist(back), len(text)

def via_csv():
    buf = io.BytesIO()
    pacsv.write_csv(table, buf)
    size = buf.tell()
    buf.seek(0)
    return pacsv.read_csv(buf), size

def via_arrow_ipc():
    sink = pa.BufferOutputStream()
    with pa.ipc.new_stream(sink, table.schema) as w:
        w.write_table(table)
    data = sink.getvalue()
    return pa.ipc.open_stream(data).read_all(), data.size

print(f"{n:,} orders, {table.nbytes / 1e6:.1f} MB in Arrow memory\n")
print(f"{'route between two tools':24s} {'bytes':>9s} {'time':>8s}   result")
for name, f in [("Python rows (a driver)", via_rows), ("JSON lines", via_json),
                ("CSV", via_csv), ("Arrow IPC stream", via_arrow_ipc)]:
    (back, size), sec = timed(f)
    same = back.equals(table)
    note = "identical" if same else "CHANGED: " + ", ".join(
        f"{c} {table.schema.field(c).type} -> {back.schema.field(c).type}"
        + (f" ({back[c].null_count:,} nulls)" if back[c].null_count else "")
        for c in table.column_names if not back[c].equals(table[c]))
    print(f"{name:24s} {'-' if size is None else f'{size / 1e6:.1f} MB':>9s} {sec * 1000:6.0f} ms   {note}")

ids = table.column("order_id").chunk(0)
view = ids.to_numpy(zero_copy_only=True)
print(f"\nNumPy view of order_id shares Arrow's buffer: {view.ctypes.data == ids.buffers()[1].address}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

On the machine that tested this lesson, the Arrow stream took about 1 millisecond, CSV about 25, Python rows 200 to 350 and JSON lines 700 to 800 (your browser will be slower): two to three orders of magnitude between the fastest and slowest route for the same table. The text formats also changed the data. JSON has no timestamp type, so ordered_at came back as strings; CSV kept timestamps but changed their unit; and NA — Namibia's country code — survived here only because pyarrow's CSV reader keeps strings as they are by default, while pandas' read_csv, by default, turns it into a missing value. Arrow moved the buffers as they were: identical, and readable in place.

As of 2026, Arrow is widely used as the common format between analytical tools. Polars and Apache DataFusion use it as their native memory format; DuckDB, ClickHouse, Spark, pandas and many databases exchange data with it; ADBC (Arrow Database Connectivity) is a database API like ODBC and JDBC that returns Arrow columns instead of rows, and Arrow Flight (with Flight SQL) moves Arrow batches over the network. Around it, a broader movement towards composable data systems — argued in a 2023 VLDB paper, the Composable Data Management System Manifesto, and built in libraries such as DataFusion and Meta's Velox — reuses query engines, plan formats (Substrait) and file readers across products.

Its limits are as real:

  • Zero-copy stops at the process. Between processes on one machine, Arrow can share memory; across a network, batches are still serialised and sent, cheaply but not for free.
  • Not every engine computes in Arrow. DuckDB, ClickHouse, Velox and most warehouses have their own internal vector formats and convert at the boundary. The conversion is cheap; the claim that "everything is Arrow inside" is not accurate.
  • The format evolves. Newer layouts such as string and list "views" (from the designs of the Umbra and Velox engines) are faster for some operations, and not every library supports every layout, so interchange can still need conversion between Arrow dialects.
  • It is not storage. Arrow IPC files are fast to read but large; Parquet remains the format for data at rest.
ExplanationH

Embedded analytics

An embedded analytical engine is a library that runs inside your program, with no server, as SQLite does for transactions. Analytical databases introduced DuckDB, Polars, DataFusion and chDB (an embedded ClickHouse), and Distributed processing measured one against a cluster and set out when one machine is enough. What is new is where such engines now run:

  • in notebooks and scripts, querying Parquet files, table formats and data frames directly;
  • in tests and CI, so that SQL transformations are checked on small fixtures without a warehouse;
  • inside serverless functions and applications, computing aggregates near the data;
  • in the browser, compiled to WebAssembly — DuckDB has an official WebAssembly build, and this course's own Python runs the same way;
  • inside other databases, as extensions that give PostgreSQL or other systems a columnar engine for analytical queries.

Their limits follow from the same design. One process on one machine: memory, disk and cores bound the work, although spilling to disk stretches it. Concurrency is limited: typically one process writes a database file at a time. There is no server to enforce access rules, audit queries or serve hundreds of dashboard users; whoever holds the files holds the data. And copying production data to laptops for convenience multiplies copies to secure, keep current and delete — a governance problem, often a privacy problem, before it is a technical one.

So the landscape has three kinds of engine rather than one winner: warehouses and lakehouse platforms for shared, governed, concurrent use; distributed engines for jobs larger than a machine; embedded engines for single-user, single-machine work and as components inside other products. Many organisations use all three on the same tables, which open formats make possible.

ExplanationH

Open tables and catalogues, as of 2026

Lakehouses and table formats explained how Iceberg, Delta Lake and Hudi add transactions and snapshots to files on object storage. The state of play by late 2026, briefly:

  • Apache Iceberg has the broadest engine support: Spark, Trino, Flink, DuckDB, the major warehouses (which can now read and, increasingly, write Iceberg tables), and object stores that offer managed Iceberg tables themselves. Its format version 3, now complete, adds binary deletion vectors, a semi-structured variant type, geospatial types and row lineage.
  • Delta Lake, from Databricks and now a Linux Foundation project, remains central to the Databricks platform and has many open-source readers. Its UniForm feature writes Iceberg-compatible metadata alongside Delta's, and Apache XTable translates metadata between formats.
  • Apache Hudi, from Uber, remains strong for record-level upserts and incremental processing; Apache Paimon, from the Flink community, targets streaming updates into lake tables. DuckDB's developers proposed DuckLake (2025), which keeps table metadata in an SQL database rather than in files.

The formats are converging in capability and interoperating more, and the competition has moved up a layer, to the catalogue: the service that names tables, coordinates commits and enforces access. Iceberg's REST catalogue protocol is the common interface; implementations include Apache Polaris (begun at Snowflake), Unity Catalog (open-sourced by Databricks), Apache Gravitino, Lakekeeper, Project Nessie, AWS Glue and Google's BigLake metastore. Whoever runs the catalogue controls governance, so choosing one is a long-term decision; prefer one that speaks the open protocol, so that engines remain swappable.

ComputationC

Zero-ETL and federated queries

Two ideas promise to remove pipelines. Zero-ETL usually means managed replication: the database vendor copies changes from an operational database into its own warehouse continuously, by change data capture (Stream processing), with no pipeline code to write. AWS's zero-ETL integrations, Microsoft Fabric's mirroring and Google's Datastream into BigQuery are examples, and other platforms offer similar features. Federated queries leave data where it is and query it in place from one engine: Trino and Presto, Spark's data sources, PostgreSQL's foreign data wrappers (the SQL standard's SQL/MED), DuckDB's scanners for other databases, and the warehouses' external tables all do this.

Both are useful, and both are named more boldly than they deserve. Replication removes the extract and load code but not the transform: the copy has the source's schema, its quirks and its mistakes, and still needs modelling, testing and documentation. Federation moves the work to query time: every query loads the source system, depends on its availability and pays its network costs. Whether either is cheap depends on pushdown — how much of the query the engine can send to each source so that less data travels. The block federates a question across an operational database (SQLite) and a lake file (Parquet), three ways.

python
import io
import sqlite3
import numpy as np
import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq

rng = np.random.default_rng(8)
days = [f"2026-{m:02d}-{d:02d}" for m in range(1, 7) for d in range(1, 29)]

# Source 1, an operational database (SQLite standing in for PostgreSQL or MySQL).
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE customers (id INTEGER PRIMARY KEY, country TEXT, segment TEXT)")
db.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, day TEXT, status TEXT, cents INTEGER)")
countries = ["Kenya", "Peru", "Vietnam", "Poland", "Ghana", "Chile"]
segments = ["school", "university", "business", "home"]
db.executemany("INSERT INTO customers VALUES (?, ?, ?)",
               [(i, countries[c], segments[s]) for i, (c, s) in
                enumerate(zip(rng.integers(0, 6, 20_000), rng.integers(0, 4, 20_000)))])
m = 150_000
db.executemany("INSERT INTO orders VALUES (?, ?, ?, ?, ?)",
               [(i, int(c), days[d], "paid" if p > 0.1 else "refunded", int(a)) for i, (c, d, p, a) in
                enumerate(zip(rng.integers(0, 20_000, m), np.sort(rng.integers(0, len(days), m)),
                              rng.random(m), rng.integers(500, 20_000, m)))])

# Source 2, the lake: website sessions in a Parquet file, written in day order in 20 row groups.
n = 600_000
lake = io.BytesIO()
pq.write_table(pa.table({"customer_id": rng.integers(0, 20_000, n),
                         "day": np.array(days)[np.sort(rng.integers(0, len(days), n))],
                         "pages": rng.integers(1, 30, n),
                         "device": rng.choice(["phone", "laptop", "tablet"], n)}),
               lake, row_group_size=30_000)
meta = pq.ParquetFile(lake).metadata

# The question: paid revenue and number of sessions per country, school customers, March 2026.
LO, HI = "2026-03-01", "2026-03-28"

def fetch(strategy):
    """Fetch from both sources; return (customer -> country, customer -> revenue, session customers, rows moved)."""
    if strategy == "pull everything":
        customers = db.execute("SELECT * FROM customers").fetchall()
        orders = db.execute("SELECT * FROM orders").fetchall()
        sessions = pq.read_table(lake)
        moved = (len(customers) + len(orders), sessions.num_rows)
        school = {i: c for i, c, s in customers if s == "school"}
        revenue = {}
        for _, cid, day, status, cents in orders:
            if cid in school and status == "paid" and LO <= day <= HI:
                revenue[cid] = revenue.get(cid, 0) + cents
        sessions = sessions.filter((pc.field("day") >= LO) & (pc.field("day") <= HI))
    else:
        school = dict(db.execute("SELECT id, country FROM customers WHERE segment = 'school'"))
        if strategy == "push filters and columns":
            orders = db.execute("SELECT customer_id, cents FROM orders WHERE status = 'paid' "
                                "AND day BETWEEN ? AND ?", (LO, HI)).fetchall()
            revenue = {}
            for cid, cents in orders:
                if cid in school:
                    revenue[cid] = revenue.get(cid, 0) + cents
        else:   # push the join and the aggregation into the database as well
            orders = db.execute("SELECT o.customer_id, SUM(o.cents) FROM orders o JOIN customers c "
                                "ON c.id = o.customer_id WHERE c.segment = 'school' AND o.status = 'paid' "
                                "AND o.day BETWEEN ? AND ? GROUP BY o.customer_id", (LO, HI)).fetchall()
            revenue = dict(orders)
        # A file cannot run a query: the reader prunes row groups by their statistics and reads two columns.
        keep = [g for g in range(meta.num_row_groups)
                if meta.row_group(g).column(1).statistics.max >= LO and meta.row_group(g).column(1).statistics.min <= HI]
        sessions = pq.ParquetFile(lake).read_row_groups(keep, columns=["customer_id", "day"])
        moved = (len(school) + len(orders), sessions.num_rows)
        sessions = sessions.filter((pc.field("day") >= LO) & (pc.field("day") <= HI))
    return school, revenue, sessions.column("customer_id").to_pylist(), moved

answers = {}
for strategy in ["pull everything", "push filters and columns", "push the aggregation too"]:
    school, revenue, session_customers, moved = fetch(strategy)
    result = {}                                                   # the final join, done by the engine
    for cid, cents in revenue.items():
        result.setdefault(school[cid], [0, 0])[0] += cents
    for cid in session_customers:
        if cid in school:
            result.setdefault(school[cid], [0, 0])[1] += 1
    answers[strategy] = dict(sorted(result.items()))
    print(f"{strategy:25s} moved {moved[0]:7,} rows from the database and {moved[1]:7,} rows from the lake")

final = answers["pull everything"]
print("all three strategies agree:", all(a == final for a in answers.values()))
for country, (cents, count) in final.items():
    print(f"  {country:8s} paid revenue {cents / 100:10,.2f}   sessions {count:5,}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

All three strategies give the same answer; what differs is the data moved, the load on the operational database, and — across regions or clouds — the egress bill. Real federated engines decide pushdown with their optimisers, and capabilities vary by source and connector, so check the query plan. Federation suits occasional questions across systems and exploring before building a pipeline; replication into analytical storage suits heavy, repeated workloads. Neither removes the need to agree what the data means.

ComputationC

Semantic layers and metrics

Ask three teams for last quarter's "revenue" and you may get three numbers: one includes refunds, one counts orders by ship date, one converts currencies at a different rate. A semantic layer (or metrics layer) declares each metric and dimension once — its formula, filters, grain and how tables join — and generates the SQL for every tool that asks, so that a dashboard, a notebook and an AI assistant answering a question in plain language all compute "revenue" the same way.

Definition— Metric types

An additive metric (revenue, units) can be summed across any dimension. A ratio metric (average order value, conversion rate) is a quotient of two additive metrics and must be computed as the ratio of their sums at the requested level, never as an average of ratios. A metric's grain is the table level at which it is defined (one row per order, one row per order line); splitting it by a dimension of a finer grain duplicates it — a fan-out.

The block declares five metrics and three dimensions over orders and order lines in SQLite, generates SQL from requests, and shows the two classic errors it prevents.

python
import sqlite3
import numpy as np

rng = np.random.default_rng(12)
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE orders (order_id INTEGER PRIMARY KEY, region TEXT, month TEXT, status TEXT, amount_cents INTEGER);
CREATE TABLE order_items (order_id INTEGER, product TEXT, qty INTEGER, item_cents INTEGER);
""")
regions = ["East Africa", "South Asia", "Andes", "Central Europe"]
products = ["atlas", "microscope", "abacus", "telescope", "globe"]
price = {"atlas": 2500, "microscope": 9000, "abacus": 800, "telescope": 15000, "globe": 3000}
items = []
for oid in range(5_000):
    region = regions[rng.choice(4, p=[0.4, 0.3, 0.2, 0.1])]
    lines = [(str(p), int(rng.integers(1, 4))) for p in rng.choice(products, rng.integers(1, 4), replace=False)]
    if region == "Central Europe":                       # a few big orders from schools
        lines = [(p, q * 5) for p, q in lines]
    total = sum(price[p] * q for p, q in lines)
    status = "paid" if rng.random() > 0.08 else "cancelled"
    db.execute("INSERT INTO orders VALUES (?, ?, ?, ?, ?)",
               (oid, region, f"2026-0{1 + oid % 3}", status, total))
    items += [(oid, p, q, price[p] * q) for p, q in lines]
db.executemany("INSERT INTO order_items VALUES (?, ?, ?, ?)", items)

# The semantic layer: each metric and dimension declared ONCE, with the grain it lives at.
METRICS = {
    "revenue":      {"grain": "order", "sql": "SUM(o.amount_cents) / 100.0", "where": "o.status = 'paid'"},
    "orders":       {"grain": "order", "sql": "COUNT(*)", "where": "o.status = 'paid'"},
    "item_revenue": {"grain": "item", "sql": "SUM(i.item_cents) / 100.0", "where": "o.status = 'paid'"},
    "units":        {"grain": "item", "sql": "SUM(i.qty)", "where": "o.status = 'paid'"},
    "aov":          {"ratio": ("revenue", "orders")},   # average order value: a ratio of two sums
}
DIMENSIONS = {"region": ("order", "o.region"), "month": ("order", "o.month"), "product": ("item", "i.product")}
FROM = {"order": "orders o", "item": "order_items i JOIN orders o ON o.order_id = i.order_id"}

def query(metric_names, by):
    parts = set()
    for m in metric_names:                       # a ratio needs its numerator and denominator
        parts.update(METRICS[m]["ratio"] if "ratio" in METRICS[m] else [m])
    grains = {METRICS[m]["grain"] for m in parts}
    if len(grains) > 1:
        raise ValueError(f"metrics {sorted(parts)} live at different grains: query them separately")
    grain = grains.pop()
    wheres = {METRICS[m]["where"] for m in parts}
    if len(wheres) > 1:
        raise ValueError(f"metrics {sorted(parts)} use different filters: query them separately")
    for d in by:
        if DIMENSIONS[d][0] == "item" and grain == "order":
            raise ValueError(f"cannot split order-level metrics by {d!r}: each order would be counted "
                             f"once per item (fan-out). Use an item-level metric instead.")
    dims = [DIMENSIONS[d][1] for d in by]
    sql = (f"SELECT {', '.join(dims + [METRICS[m]['sql'] for m in sorted(parts)])} FROM {FROM[grain]} "
           f"WHERE {wheres.pop()}" + (f" GROUP BY {', '.join(dims)}" if dims else ""))
    rows = []
    for r in db.execute(sql):
        vals = dict(zip(sorted(parts), r[len(dims):]))
        out = list(r[:len(dims)])
        for m in metric_names:
            a, b = METRICS[m].get("ratio", (m, None))
            out.append(vals[a] / vals[b] if b else vals[m])
        rows.append(tuple(out))
    return rows

print("revenue, orders and average order value by region:")
by_region = query(["revenue", "orders", "aov"], ["region"])
for region, rev, n, aov in by_region:
    print(f"  {region:15s} {rev:12,.2f} {n:6,}  AOV {aov:8.2f}")
(total_rev, total_n, total_aov), = query(["revenue", "orders", "aov"], [])
print(f"  {'all regions':15s} {total_rev:12,.2f} {total_n:6,}  AOV {total_aov:8.2f}")
print(f"  the average of the four regional AOVs would say {np.mean([r[3] for r in by_region]):.2f}")

naive = db.execute("""SELECT i.product, SUM(o.amount_cents) / 100.0 FROM orders o
                      JOIN order_items i ON i.order_id = o.order_id WHERE o.status = 'paid'
                      GROUP BY i.product""").fetchall()
print(f"\nhand-written 'revenue by product' joins orders to items: total {sum(r[1] for r in naive):,.2f}"
      f" ({sum(r[1] for r in naive) / total_rev:.2f} x the real revenue)")
try:
    query(["revenue"], ["product"])
except ValueError as err:
    print("semantic layer refuses:", err)
print("item-level revenue by product instead:")
for product, rev, units in query(["item_revenue", "units"], ["product"]):
    print(f"  {product:11s} {rev:12,.2f}  {units:5,} units")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The true average order value is 342.42. Averaging the four regional values gives 485.29, because Central Europe's few, very large school orders count as much as East Africa's 1,807 orders: the average of ratios answers no one's question. The hand-written "revenue by product" joined each order to its lines and so counted an order with three products three times, reporting 2.33 times the real revenue; the layer, knowing that revenue lives at order grain and product at line grain, refuses, and the line-level metric adds up exactly to 1,576,485.00.

As of 2026, semantic layers come in several forms: modelling languages inside BI tools (Looker's LookML, Power BI's semantic models), standalone layers such as Cube (open source) and dbt's MetricFlow, and definitions stored in warehouses and catalogues themselves (Snowflake's semantic views, Databricks' metric views). Vendors announced an Open Semantic Interchange effort in 2025 to exchange definitions between them; whether it, or any one approach, becomes the standard is open. The interest has grown with AI assistants that write SQL from questions, because a model given declared metrics and join paths has far less room for silent errors than one guessing from raw table names.

Worked examplesH

Cost management

Separated storage and compute made capacity elastic, and with consumption pricing, made spending elastic too. Analytical databases covered the cost of a single query; the Cloud Computing elective's cost and FinOps covers cloud spending in general. For a data platform, the recurring drivers are these:

  • Idle compute: clusters and warehouses left running between jobs.
  • Repeated work: dashboards refreshing far more often than anyone looks, and several teams computing the same tables.
  • Storage that never ages: every snapshot, copy and intermediate table kept forever at the hottest price.
  • Data movement: egress between regions and clouds, which federation and multi-cloud designs can multiply.
  • Small files and metadata: thousands of tiny files that slow every query and cost requests (compaction, in Lakehouses and table formats, is the cure).

Example— Two quiet line items

Prices here are hypothetical round numbers; real ones vary by provider, region and contract. A warehouse cluster billed at USD 8 an hour runs all month because nobody set it to suspend: 24×30=72024 \times 30 = 720 hours, USD 5,760. The jobs that use it run 4 hours a day, so with automatic suspension it would cost 4×30×8=4 \times 30 \times 8 = USD 960. Separately, a nightly federated query pulls 2 TB from a database in another region at a hypothetical egress price of USD 0.02 per GB: 2,000×0.02=2{,}000 \times 0.02 = USD 40 a night, about USD 1,200 a month — for a table that changes by a few gigabytes a day and could be replicated incrementally instead.

The practices that control these are mostly organisational. Attribute every query and job to a team, through labels, tags or separate accounts, and show each team its costs; set budgets and alerts; make auto-suspend and storage lifecycle rules (move old data to cheaper tiers, expire old snapshots) the default; fingerprint queries to find the few repeated ones that dominate the bill (one exercise builds this). The FinOps Foundation's FOCUS specification gives cost data from different providers a common schema. Regulation is moving too: the EU's Data Act requires cloud providers to phase out charges for switching to another provider, by January 2027.

History & global context

What changed since Hadoop, and what did not

Hadoop era (about 2010–2015)2026
StorageHDFS on the cluster's own disksobject storage, separate from compute
TablesHive tables: folders plus a metastoreopen table formats with transactions and snapshots
ComputeMapReduce, then Hive on Tez and Sparkvectorised SQL engines, Spark, embedded engines
ResourcesYARN on a fixed clusterKubernetes, serverless and per-second billing
Data filestext, SequenceFiles, then ORC and Parquetmostly Parquet; Arrow in memory
TransformationJava jobs, Pig scripts, ETL toolsSQL models in version control (ELT)
Who operates ita platform team running the clustermostly managed services, plus a platform team

What changed is mostly the packaging: storage separated from compute, managed services replaced self-operated clusters, columnar formats and vectorised engines made one machine do what once took dozens, and open interfaces let pieces be swapped. Hadoop clusters still run in many banks, telecoms and public bodies, especially where data must stay on premises, and HDFS's ideas live on in object storage.

What did not change is most of this elective. Shuffles still cost what they cost, and skew still makes one task the slowest (Batch processing and MapReduce). The small-files problem moved from HDFS's NameNode to object-store listings and table metadata. Partitioning, data modelling with a declared grain, idempotent and re-runnable pipelines, data quality, lineage, access control and privacy are as necessary as in 2012 — and the hardest problems are still agreeing what the data means and who owns it. Each era also had its hype: "big data" for everything in 2012, "the modern data stack" of a dozen tools in 2021, and "AI-ready data" in 2026. In each, the teams that did well chose the fewest tools that did the job, and kept the fundamentals.

Common misconceptionsH

Common misconceptions

Misconception

"Zero-ETL means no data engineering." It means the extract-and-load step is managed for you. The replicated tables still need modelling, testing, documentation and cost control, and the replication itself must be monitored.

Misconception

"Arrow makes everything zero-copy." Within one process, and between processes sharing memory, it can. Across a network, or into an engine with its own internal format, data is still converted or serialised — far more cheaply than through rows or text.

Misconception

"An embedded engine can replace the warehouse." For single-user work on one machine's worth of data it often can. For shared, governed, concurrent use — access control, auditing, hundreds of dashboard users — it is a component, not a replacement.

Misconception

"Open formats remove lock-in." They remove lock-in at the file layer. Catalogues, SQL dialects, semantic definitions, pipelines and people's skills can still tie an organisation to a vendor; open formats make leaving possible, not free.

Exercises

Exercises

1. The modern data stack in 2026

Warm-upHuman

An orders table has three paid orders: A for 100, B for 50 and C for 80. The order_items table has three lines for order A, one for B and two for C. An analyst writes

SELECT SUM(o.amount) FROM orders o JOIN order_items i ON i.order_id = o.order_id;

What number does the query return?

2. The modern data stack in 2026

CoreComputational

A daily summary table has one row per day and region: (week, region, revenue, orders). Write weekly_aov(daily), returning a dictionary that maps each (week, region) to its average order value for the week — or None if the region had no orders that week.

The starter averages the daily averages (and treats days without orders as an average of 0). Fix it.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

3. The modern data stack in 2026

StretchComputational

A warehouse bills USD usd_per_tb per terabyte (101210^{12} bytes) scanned. Write query_costs(log, usd_per_tb) for a query log, a list of dicts with keys team, sql and bytes. Return a tuple of three things:

  1. a dict of total cost per team, each rounded to 2 decimals;
  2. the fingerprint whose queries cost the most in total;
  3. that fingerprint's total cost, rounded to 2 decimals.

A query's fingerprint is its SQL lowercased, with every string literal ('...') and every number replaced by ?, and every run of whitespace collapsed to one space, with no leading or trailing space. So SELECT * FROM t WHERE day = '2026-03-01' and select * from t where day = '2026-03-02' share the fingerprint select * from t where day = ?.

The starter forgets to convert bytes to terabytes and ranks single queries instead of fingerprints.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

4. The modern data stack in 2026

CoreHuman

A data lake holds 300 TB: 30 TB written in the last 90 days, 90 TB between 90 days and a year old, and 180 TB older than a year. Using hypothetical monthly prices per TB of USD 20 (hot tier), USD 10 (infrequent-access tier) and USD 2 (archive tier), how many USD per month does a lifecycle policy save that keeps the newest data hot, moves data older than 90 days to infrequent access, and archives data older than a year — compared with keeping all 300 TB hot?

5. The modern data stack in 2026

CoreHuman

Which statements about Apache Arrow are accurate as of 2026? Select all that apply.

Choose all that apply

6. The modern data stack in 2026

CoreHuman

A database vendor offers "zero-ETL" replication from its operational database into its warehouse. Which description is the most accurate?

Choose one

7. The modern data stack in 2026

StretchHuman

A public-transport authority keeps 2 TB of ticketing data as Iceberg tables on object storage. It needs (1) dashboards for 300 staff, with row-level access rules by region, and (2) an analyst's weekly exploration of a 5 GB extract of anonymised journeys. Which plan fits best?

Choose one

Further reading and resources