Skip to content
LumenaryAcademy

Lumenary CS · Electives

Data Engineering & Big Data

A Lumenary CS elective: the systems that move, store and process data at every scale — data models and formats, analytical databases, batch and distributed processing, pipelines, streaming, lakehouses, data quality and governance, and data for AI.

This elective of Lumenary CS teaches the engineering behind every dashboard, report and AI model: getting data from where it is produced to where it is useful, reliably and at scale. It goes deeper than Lumenary CS's storage at scale and batch and stream processing lessons, from file formats and columnar engines through MapReduce and Spark, orchestration and streaming, to lakehouses, data quality, governance and the data pipelines behind large models.

It assumes Lumenary CS up to its Scale level and comfort with SQL (the SQL course teaches it). Code is in Python and SQL, and every block with a Run in browser button runs there, with nothing to install: you will write a MapReduce engine, partition and shuffle data, simulate a stream with watermarks and build a tiny lakehouse table. Code for Spark, Kafka, Airflow and cloud warehouses is shown and explained, not run.

It is written to inform, not to sell: tools and companies are named fairly and by link, and the lessons are as plain about costs, failures and trade-offs as about what works. Every lesson was drafted with AI assistance and is awaiting review by a human expert. The text, exercises and code are original and licensed CC BY-SA 4.0 (code Apache-2.0). Papers, specifications and projects' own documentation are linked, not retold.

AI-draftedAwaiting review by a data engineer
  • Chloe · The Computer Scientist
    “What's the idea that makes it work?”
  • Margaret · The Experimenter
    “What happens when we run it?”
  • Gideon · The Verifier
    “Is it correct in every case?”

Foundations · 4 lessons

Data systems

The building blocks: what data engineers do, data models and file formats, SQL and analytical databases, and batch processing from MapReduce onwards.

  1. 1What data engineers doFrom the place data is born to the place it is useful — the lifecycle, the trade-offs and the people. 60 min
  2. 2Data models and file formatsHow to shape data for the questions people ask, and how to lay it out on disk and in memory so that answering them is cheap. 80 min
  3. 3Analytical databasesWhy the engines that answer questions over billions of rows look so different from the database behind a shop's checkout — and what their answers cost. 80 min
  4. 4Batch processing and MapReduceThe 2004 idea that made thousand-machine jobs routine — build a small engine, break it in instructive ways, and see why Spark came next. 80 min

Practice · 5 lessons

Pipelines at scale

Running data systems for real: distributed processing with Spark, pipelines and orchestration, stream processing, lakehouses and open table formats, and data quality and governance.

  1. 5Distributed processing with SparkHow a dataflow engine plans a job, cuts it into stages and runs it on many machines — and when one machine does the job better. 85 min
  2. 6Pipelines and orchestrationTurning a pile of scripts into a pipeline that runs in the right order, recovers from failure, rebuilds the past on demand and tells you when it is late. 80 min
  3. 7Stream processingComputing over data that never stops arriving — logs and consumer groups, event time and watermarks, results that are exactly right after crashes, and capturing every change a database makes. 85 min
  4. 8Lakehouses and open table formatsHow a folder of Parquet files in object storage becomes a table with transactions, snapshots, time travel and safe schema changes — and the housekeeping that keeps it fast. 85 min
  5. 9Data quality and governanceMaking data trustworthy and keeping it in good hands — contracts, tests, monitors, lineage, catalogs, access control, privacy law, and deletion that actually deletes. 85 min

Frontier · 3 lessons

Data in 2026

Where data engineering is going: data for AI, the modern data stack, and big data in science and society.

  1. 10Data for AIThe data engineering inside machine learning — training corpora built from the web, features served without leaking the future, vectors searched approximately, and evaluation sets kept honest. 90 min
  2. 11The modern data stack in 2026How the pieces of a data platform fit together today — shared in-memory formats, embedded engines, open tables, zero-ETL and federation, semantic layers and the bill — and what is genuinely new since Hadoop. 85 min
  3. 12Big data in science and societyParticle physics, radio astronomy, genomics, climate and Earth observation, web-scale logs, official statistics and public health — what each one's data problem is, and the engineering idea it teaches. 85 min

Further reading and resources