Where the big data really is
"Big data" is often pictured as a technology company's clickstream. Some of the largest, oldest and most carefully engineered data systems belong instead to science and to the public sector: a particle detector that sees forty million collisions a second, a telescope array spread across two continents, sequencers that read a human genome in a day, weather and climate models that cover the globe hour by hour, and the national systems that count a country's people and its illnesses. They faced the problems of this elective — volume, velocity, formats, quality, privacy, cost — years before most companies did, and they publish how they solved them.
This lesson visits six such systems, each described from its own published sources, and draws out the engineering idea it teaches best. Three of the ideas get a small simulation: a particle-physics trigger, the chunking of climate data, and the correction of a disease count for reporting delays.
Particle physics: deciding in microseconds
At the Large Hadron Collider, bunches of protons cross every 25 nanoseconds — 40 million times a second — inside detectors such as ATLAS and CMS, each crossing producing tens of overlapping collisions and about a megabyte of detector readout. Recording everything would mean tens of terabytes a second, and the interesting events (a Higgs boson, a hint of new physics) are rarer than one in a billion. So the experiments decide, as the data arrives, what to keep (CERN's account).
The level-1 trigger is custom electronics that examines coarse energy deposits and muon hits and decides within a few microseconds, while the full readout waits in buffers; it keeps about 100,000 crossings a second. The high-level trigger is software on a farm of servers (CMS added GPUs to its farm for Run 3) that reconstructs each surviving event in milliseconds and keeps about a thousand to a few thousand a second. Other experiments chose differently: in Run 3 LHCb removed its hardware trigger and reconstructs all 30 million crossings a second in software, the first stage on GPUs. What is kept goes to CERN's data centre and out through the Worldwide LHC Computing Grid — over 170 sites in 42 countries, with about 1.4 million processor cores and 1.5 exabytes of storage as CERN describes it — in a tiered federation of storage and processing. Data and analysis tools are released through the CERN Open Data portal.
A trigger menu is a list of paths — "an energetic electron", "two muons", "a lot of missing energy" — each with a threshold and a share of the rate budget. The block simulates a two-level menu on toy events: ordinary collisions, a heavy particle decaying to energetic objects, and a long-lived particle that deposits little energy but decays away from the beam line. Thresholds are set so that background exactly fills each level's budget.
import numpy as np
rng = np.random.default_rng(2026)
CROSSINGS = 40e6 # bunch crossings per second at the LHC
L1_BUDGET, HLT_BUDGET = 100e3, 1e3 # crossings per second each stage may accept
def events(kind, n):
"""Toy events: true transverse energy (GeV) and the largest track displacement d0 (mm)."""
if kind == "background": # ordinary collisions: mostly soft; a few displaced tracks
et = rng.exponential(12.0, n) + rng.exponential(4.0, n)
d0 = np.where(rng.random(n) < 0.0001, rng.exponential(1.5, n), np.abs(rng.normal(0, 0.03, n)))
elif kind == "heavy": # a heavy particle decaying to energetic objects
et = rng.normal(150, 25, n)
d0 = np.abs(rng.normal(0, 0.03, n))
else: # a long-lived particle: soft, but decays away from the beam line
et = rng.normal(30, 8, n)
d0 = rng.exponential(6.0, n) + 1.0
return {"d0": d0,
"et_l1": et * (1 + 0.25 * rng.normal(size=n)), # level 1: coarse, within microseconds
"et_hlt": et * (1 + 0.05 * rng.normal(size=n)), # high-level trigger: near-offline precision
"track_l1": rng.random(n) < np.where(d0 > 2.0, 0.6, 0.0002)} # coarse displaced-track flag
bg = events("background", 2_000_000) # 50 ms of collisions
signals = {"heavy": events("heavy", 20_000), "long-lived": events("long-lived", 20_000)}
def lowest_cut(rate, budget):
"""Bisect for the lowest energy cut whose background rate fits the budget (rate falls as the cut rises)."""
lo, hi = 0.0, 1000.0
for _ in range(40):
mid = (lo + hi) / 2
lo, hi = (lo, mid) if rate(mid) <= budget else (mid, hi)
return hi
def design(with_track_paths):
l1_extra = (lambda e: e["track_l1"]) if with_track_paths else (lambda e: False)
hlt_extra = (lambda e: (e["d0"] > 2.0) & (e["et_hlt"] > 25)) if with_track_paths else (lambda e: False)
c1 = lowest_cut(lambda c: np.mean((bg["et_l1"] > c) | l1_extra(bg)) * CROSSINGS, L1_BUDGET)
l1 = lambda e: (e["et_l1"] > c1) | l1_extra(e)
c2 = lowest_cut(lambda c: np.mean(l1(bg) & ((bg["et_hlt"] > c) | hlt_extra(bg))) * CROSSINGS, HLT_BUDGET)
hlt = lambda e: l1(e) & ((e["et_hlt"] > c2) | hlt_extra(e))
return c1, c2, l1, hlt
for with_track in [False, True]:
c1, c2, l1, hlt = design(with_track)
print(("menu 2: energy OR displaced-track paths" if with_track else "menu 1: energy paths only")
+ f"; energy cuts L1 > {c1:.0f} GeV, HLT > {c2:.0f} GeV")
print(f" background: L1 {np.mean(l1(bg)) * CROSSINGS / 1e3:5.1f} kHz, "
f"HLT {np.mean(hlt(bg)) * CROSSINGS:5.0f} Hz")
for name, s in signals.items():
print(f" {name:10s} kept by L1 {np.mean(l1(s)):6.1%}, by both {np.mean(hlt(s)):6.1%}")
print(f"\nkept: 1 crossing in {CROSSINGS / HLT_BUDGET:,.0f}. At 1 MB per event, "
f"{CROSSINGS * 1e6 / 1e12:.0f} TB/s arrive and {HLT_BUDGET * 1e6 / 1e9:.0f} GB/s are stored.")The engineering idea is reduce at the source, with decisions you can measure. Rejected data is gone, so the experiments keep "zero-bias" and loose "minimum-bias" samples at tiny prescaled rates, measure each trigger's efficiency with independent triggers, and keep scouting streams that store a compact summary of many more events than could be stored in full. The High-Luminosity LHC, planned to start around 2030 with several times more collisions per crossing, raises the planned first-level output to about 0.75–1 MHz and the stored rate to several thousand a second or more.
Radio astronomy: the array as a computer
The SKA Observatory, an intergovernmental organisation with headquarters at Jodrell Bank in the United Kingdom, is building two radio telescopes: SKA-Low, 131,072 antennas grouped in 512 stations at Inyarrimanha Ilgari Bundara, CSIRO's Murchison Radio-astronomy Observatory in Western Australia, on Wajarri Yamaji Country; and SKA-Mid, 197 dishes in the Karoo region of South Africa, including the 64 of the existing MeerKAT telescope. Construction began in 2021, with science operations to grow through the late 2020s.
An interferometer is a data system as much as an instrument: the sky image is computed from correlations between the signals of every pair of receivers. Raw antenna voltages arrive far too fast to store, so the data is reduced in stages near the telescopes and then processed into science products in Science Data Processor facilities in Perth and Cape Town.
On average about 8 terabits a second will flow from the telescopes into the central signal processors that correlate them, and SKAO expects to archive over 700 petabytes a year — too much to send every astronomer a copy. The plan is a federation: an SKA Regional Centre Network of computing centres in member countries where astronomers analyse the data in place — the same shape as the LHC grid. The precursor telescopes (MeerKAT in South Africa, ASKAP and the Murchison Widefield Array in Australia) and pathfinders such as LOFAR in Europe have been testing these ideas for a decade.
Two further ideas come from radio astronomy. A transient buffer keeps the most recent raw data for a short time, so that when the pipeline detects a fast radio burst — a millisecond flash from another galaxy — the raw data around it can be saved; Canada's CHIME telescope does the same with its baseband data. And the Event Horizon Telescope, which imaged the black holes in M87 and the Milky Way, recorded petabytes on hard drives at telescopes around the world and flew the drives to its correlators, because no network link from a mountaintop or the South Pole could move that much data as fast. Physical shipment remains the highest-bandwidth link for one-off transfers, which is why cloud providers rent out storage appliances for the same purpose.
Genomics: data about people that cannot be anonymised
The cost of sequencing a human genome fell from about USD 100 million in 2001 to under USD 1,000 by the early 2020s, according to the NHGRI's series, far faster than computing costs fell. A genome sequenced to 30-fold coverage means about 93 billion bases read, with a quality score for each: on the order of 200 GB of text before compression. National programmes sequence hundreds of thousands of people; UK Biobank released whole genomes for its 500,000 participants in 2023.
Three engineering ideas stand out. Domain formats: sequence data has its own formats (FASTQ for reads, BAM and CRAM for aligned reads, VCF for variants), and CRAM compresses aligned reads by storing only their differences from a reference genome — the kind of structure-aware encoding that Parquet applies to tables. Shared infrastructure: the International Nucleotide Sequence Database Collaboration — GenBank in the United States, the European Nucleotide Archive in the United Kingdom and the DNA Data Bank of Japan — exchanges new public sequence data daily, so a submission to any one appears in all three. And the Global Alliance for Genomics and Health writes standards for retrieving data (htsget), discovering it (Beacon) and encoding who may use it for what (data-use and access "passports"). Pipelines are written in workflow languages — Nextflow, Snakemake, WDL, CWL — so that an analysis can be rerun identically elsewhere.
The deepest issue is privacy. A genome identifies its owner and reveals information about relatives who never consented. In 2013, Gymrek and colleagues showed that some "anonymous" genomes in a research database could be traced to named individuals by matching Y-chromosome markers against public genealogy websites. So the field increasingly brings analysis to the data instead of releasing it: trusted research environments where approved researchers run code on data they cannot download, with outputs checked before release — the "Five Safes" framework, from the UK's Office for National Statistics, sets out the principles (safe people, projects, settings, data and outputs). Fair representation is a concern too: most sequenced genomes so far come from people of European ancestry, which limits what findings mean for everyone else, and projects such as H3Africa and the Human Pangenome Reference Consortium work to broaden it.
Climate and Earth observation: arrays, not tables
Weather and climate data are large multidimensional arrays: a variable on a grid of latitude, longitude, height and time. ECMWF's ERA5 reanalysis for the EU's Copernicus Climate Change Service reconstructs the global atmosphere hourly from 1940 on a grid of about 31 km; the climate-model experiments of the Coupled Model Intercomparison Project (CMIP) are distributed through the Earth System Grid Federation, with data nodes on several continents. Satellites add a flood of images: the EU's Copernicus Sentinel missions and the NASA–USGS Landsat archive, free and open since 2008, observe every point on land every few days.
The classic formats — NetCDF and HDF5 in research, GRIB at weather services under the World Meteorological Organization — were designed for files on disks next to supercomputers. Cloud object storage favours formats that a reader can fetch in pieces with ranged requests: Zarr for chunked, compressed arrays, Cloud-Optimized GeoTIFF for images, and the STAC specification for catalogues of scenes. As with Parquet's row groups, the design question is the chunk shape: which neighbouring values are stored together, and so which queries are cheap. The block works it out for one ERA5-like variable.
from datetime import date
import math
# One ERA5-like variable: hourly, on a 0.25-degree global grid, 1940 to 2025, as 32-bit floats.
hours = (date(2026, 1, 1) - date(1940, 1, 1)).days * 24
lat, lon = 721, 1440
one = hours * lat * lon * 4
print(f"{hours:,} hours x {lat} x {lon} points x 4 bytes = {one / 1e12:.2f} TB for one variable at one level;")
print(f"on 37 pressure levels, {37 * one / 1e12:.0f} TB. Hundreds of variables make petabytes.\n")
layouts = { # chunk shape: (hours, latitudes, longitudes)
"map": (1, 721, 1440), # one hour of the whole globe per chunk
"series": (8760, 10, 10), # a year at 10 x 10 points per chunk
"balanced": (24, 90, 90), # a day over 90 x 90 points per chunk
}
queries = { # the box each query reads, aligned to chunk edges
"one global map": (1, 721, 1440),
"86 years at one point": (hours, 1, 1),
"a month over 10 x 10 degrees": (720, 40, 40),
}
def size(nbytes):
return f"{nbytes / 1e9:,.1f} GB" if nbytes >= 1e9 else f"{nbytes / 1e6:,.1f} MB"
print(f"{'bytes read (chunks touched)':30s}" + "".join(f"{name:>24s}" for name in layouts))
for qname, box in queries.items():
cells = []
for chunk in layouts.values():
n = math.prod(math.ceil(b / c) for b, c in zip(box, chunk)) # chunks touched
cells.append(f"{size(n * math.prod(chunk) * 4)} ({n:,})")
print(f"{qname:30s}" + "".join(f"{c:>24s}" for c in cells))
need = {q: math.prod(b) * 4 for q, b in queries.items()}
print("\nbytes actually wanted: " + "; ".join(f"{q}: {size(v)}" for q, v in need.items()))No chunk shape suits every query. Map-shaped chunks serve a global map with one 4.2 MB read, but a single point's 86-year history then touches all 753,888 chunks, 3.1 TB of reads for 3.0 MB of answer. Time-series chunks reverse the costs. Balanced chunks avoid the worst of each — 112 MB for the map, 24.4 GB for the history — and suit regional studies best. This is the row-versus-column choice of Data models and file formats in more dimensions, and data providers increasingly publish popular datasets in more than one chunking.
Two habits of this community are worth copying. They bring computation to the data: platforms such as the open Pangeo community's tools, the Copernicus Data Space Ecosystem and commercial Earth-observation clouds let users run analyses next to the archive instead of downloading it. And they decide outputs in advance: a climate model generates far more numbers than can be stored, so each experiment specifies which variables are saved at which frequency, a trigger menu for simulations. The EU's Destination Earth initiative, run by ECMWF, ESA and EUMETSAT, goes further, streaming model output to applications as it is produced.
Web-scale logs: the log as the backbone
The web companies' contribution was to make logs — every request, click and error — the raw material of analysis, and to publish the systems they built. Google's Dremel (2010) brought interactive SQL to columnar logs; Facebook's Scuba (2013) kept recent logs in memory across many servers, with sampling for the busiest streams, for analysis within seconds; LinkedIn's Kafka (2011) made a partitioned, append-only log the central bus through which all data flows, the architecture behind Stream processing.
Their lasting ideas: treat the append-only log as the source of truth and derive tables from it; accept approximate answers where they are good enough, with sampling and sketches such as HyperLogLog, which counts distinct users in a few kilobytes (Lumenary CS's randomness in algorithms); and set retention deliberately, because keeping every log for ever is expensive and, when the logs contain IP addresses or user identifiers, a liability under data-protection law. Public, aggregated releases show that usefulness and privacy can coexist: the Wikimedia Foundation publishes hourly page-view counts for every Wikipedia article, aggregated so that no reader can be identified.
Official statistics and public health: timely, complete, private
National statistical offices have measured whole populations for centuries, and they now combine surveys with administrative registers and new sources: Statistics Netherlands has used supermarket scanner data in its consumer price index since 2002, Nordic countries run censuses entirely from registers, and Indonesia's statistics office has used mobile-phone positioning data for tourism statistics. The UN's Committee of Experts on Big Data and Data Science for Official Statistics coordinates such work across countries. The public trust these offices depend on rests on confidentiality: the US Census Bureau protected the 2020 Census releases with differential privacy, a choice debated by its users and explained in Lumenary Statistics' privacy and official statistics.
Public health added urgency. During COVID-19, dashboards from national agencies, Johns Hopkins University and Our World in Data were rebuilt daily from feeds of very uneven quality, and the failures were instructive. In October 2020 nearly 16,000 positive test results in England were left out of contact tracing for about a week, because a pipeline passed data through the legacy Excel file format, whose sheets hold at most 65,536 rows, and the extra rows were silently dropped. The fix was trivial; the lesson — test the limits of every format in a pipeline, and reconcile counts between stages — is the whole of Data quality and governance.
A subtler problem is reporting delay. A case is counted by the day symptoms began, but reaches the national database days later, so the most recent days are always incomplete and an epidemic curve seems to fall at its right edge even while it is rising. Agencies such as Germany's Robert Koch Institute and the InfoGripe system at Brazil's Fiocruz publish nowcasts that correct for this. The block builds the simplest one: estimate the delay distribution from days old enough to be complete, then scale each recent day by the share of its cases expected to have arrived.
import numpy as np
rng = np.random.default_rng(31)
T = 70 # 70 days of a synthetic outbreak; today is day 69
days = np.arange(T)
expected = 40 * np.exp(0.06 * days) / (1 + np.exp(0.12 * (days - 62))) # rising, starting to level off
true_onsets = rng.poisson(expected) # cases by date of symptom onset
# Each case reaches the national database some days after onset: testing, the lab, then the report.
delay_p = np.array([0.02, 0.08, 0.15, 0.18, 0.16, 0.12, 0.09, 0.07, 0.05, 0.03, 0.02, 0.015, 0.01, 0.005])
delay_p /= delay_p.sum()
reported = np.zeros((T, len(delay_p)), dtype=int) # reported[onset day, delay]
for t in days:
reported[t] = rng.multinomial(true_onsets[t], delay_p)
# What the database shows today: only cases with onset + delay <= today.
visible = np.array([reported[t, :T - t].sum() for t in days])
# Estimate the delay distribution from onset days old enough to be complete (14+ days ago).
complete = reported[: T - len(delay_p)]
delay_hat = complete.sum(axis=0) / complete.sum()
reported_by = np.cumsum(delay_hat) # share of a day's cases reported within d days
# Nowcast: scale each recent day up by the share of its cases expected to have arrived so far.
nowcast = np.array([visible[t] / reported_by[min(T - 1 - t, len(delay_p) - 1)] for t in days])
print("onset day true visible today nowcast")
for t in range(T - 12, T):
print(f" {t:3d} {true_onsets[t]:5d} {visible[t]:5d} {nowcast[t]:7.0f}")
last7 = slice(T - 7, T)
print(f"\nlast 7 days: true {true_onsets[last7].sum()}, visible {visible[last7].sum()}, "
f"nowcast {nowcast[last7].sum():.0f}")
print(f"week-on-week change: true {true_onsets[last7].sum() / true_onsets[T - 14:T - 7].sum() - 1:+.0%}, "
f"visible {visible[last7].sum() / visible[T - 14:T - 7].sum() - 1:+.0%}, "
f"nowcast {nowcast[last7].sum() / nowcast[T - 14:T - 7].sum() - 1:+.0%}")The raw counts say cases fell 57% in a week; the truth is a 3% fall, and the nowcast says 1% up — close, and in the right regime. Nowcasting is a statistical model, treated properly in Lumenary Statistics' time series lesson; the data-engineering part is to record both dates for every case, keep every version of the data as it arrived, and never publish the right edge of a curve without saying it is incomplete.
The ideas they share
| Case | Main data problem | Engineering idea |
|---|---|---|
| LHC experiments | 40 million crossings a second, rare signals | reduce at the source in stages; measure what you discard; federate the computing |
| SKA and radio arrays | raw signals too fast to store | reduce in place; buffer and trigger on transients; ship disks when networks are too slow |
| Genomics | identifying, inherited, fast-growing data | domain formats and compression; shared standards; analysis inside trusted environments |
| Climate and Earth observation | multidimensional arrays at petabyte scale | chunk for the queries; cloud-native formats; bring compute to the data; choose outputs in advance |
| Web-scale logs | endless streams of events | the log as source of truth; sampling and sketches; deliberate retention |
| Statistics and public health | timeliness against completeness and privacy | test every format's limits; record event and report dates; nowcast; privacy-protecting release |
Read down the right-hand column and the elective's themes reappear: layouts chosen for the questions, computation moved to the data, federation instead of copies, approximate answers with known error, and data about people handled with restraint. Nothing in the column is specific to science; the scientific projects simply met the limits earlier, and wrote down what they learned.
Common misconceptions
Misconception
"Science keeps all its data." The largest scientific instruments keep a tiny, deliberately chosen fraction: the LHC experiments store about one crossing in tens of thousands, and radio arrays keep correlated products, not raw voltages. Their skill lies in choosing well and measuring the choice.
Misconception
"Removing names makes genomic or health data anonymous." Genomes, and detailed records of dates, places and conditions, can be linked back to people and their relatives. Protection comes from controlled access, aggregation and formal methods such as differential privacy, not from deleting a name column.
Misconception
"The latest numbers are the most reliable." For anything with reporting delays — cases, sales, sensor feeds with retries — the latest numbers are the least complete, and they bias towards a fall.
Exercises
1. Big data in science and society
Warm-upHumanA detector sees 40 million bunch crossings per second. Its level-1 trigger keeps 1 crossing in 400, and its high-level trigger keeps 1 in 100 of those. Each stored event is 1 MB ( bytes). How many terabytes ( bytes) are stored in 10 hours of running?
2. Big data in science and society
CoreComputationalWrite run_menu(events, paths), a trigger menu with prescales. Each path is a tuple (name, condition, prescale), where condition(event) returns True or False. A path with prescale keeps the 1st, the th, the th, … of the events that satisfy its condition, counting separately for each path (prescale 1 keeps them all). An event is accepted if at least one path keeps it.
Return a tuple: the list of accepted event indices in order, and a dict giving, for each path name, how many events that path kept.
Prescaled paths, such as a "zero-bias" path that keeps every 1,000th crossing whatever it contains, are how experiments keep an unbiased sample to measure their other triggers with.
3. Big data in science and society
CoreHumanA year of hourly data on a global grid is stored as an array of shape (8,760 hours, 721 latitudes, 1,440 longitudes) in chunks of shape (24, 90, 90). A user reads one week — 168 consecutive hours starting at midnight on the first of January — for the whole globe. How many chunks must be read?
4. Big data in science and society
CoreHumanA radio observing campaign records 5 PB ( bytes) at telescopes around the world. How many days, to the nearest whole day, would it take to send it over a dedicated 10 Gb/s link ( bits per second) running flat out with no overhead?
5. Big data in science and society
CoreComputationalWrite nowcast(visible, reported_by). visible[i] is the number of cases with onset on day i that have been reported so far; the last element is today. reported_by[d] is the share of a day's cases that are reported within d days of onset (it rises to 1.0 at its last element, and delays longer than that are treated as complete). Return a list with the estimated eventual count for every day: the visible count divided by the share expected to have arrived by today.
6. Big data in science and society
CoreHumanA research group wants to publish 2,000 whole genomes "anonymously" by removing names, dates of birth and addresses. What is the main problem?
7. Big data in science and society
StretchHumanWhich pairings of a case with the engineering idea it illustrates are accurate? Select all that apply.
Further reading and resources
- CERN: StorageCERN's own account of collision rates, triggers and the data volumes of the LHC runs.
- CERN: The Worldwide LHC Computing Grid
- CERN Open Data portalCollision data, simulations and software released by the LHC experiments for education and research.
- ROOTCERN's data-analysis framework and file format, used across particle physics.
- SKA ObservatoryThe organisation building SKA-Low in Western Australia and SKA-Mid in South Africa.
- SKAO: Big dataThe observatory's description of its data rates, Science Data Processors and regional centres.
- Event Horizon Telescope
- NHGRI: DNA sequencing costsThe US National Human Genome Research Institute's series of the cost of sequencing a genome since 2001.
- International Nucleotide Sequence Database Collaboration
- Global Alliance for Genomics and Health (GA4GH)Standards for sharing genomic and health data, including htsget, Beacon and data-use and access passports.
- Identifying Personal Genomes by Surname Inference · Melissa Gymrek, Amy L. McGuire, David Golan, Eran Halperin and Yaniv Erlich · 2013Science 339(6117). Showed that "anonymous" genomes can be re-identified from public genealogy data.
- Copernicus Climate Change Service: ERA5ECMWF's global reanalysis, hourly from 1940.
- Earth System Grid FederationThe federated archive through which CMIP climate-model output is distributed.
- ZarrA format for chunked, compressed N-dimensional arrays, designed for cloud object storage.
- SpatioTemporal Asset Catalog (STAC)
- Dremel: Interactive Analysis of Web-Scale Datasets · Sergey Melnik et al. · 2010VLDB 2010; Google's system for interactive queries over logs.
- Scuba: Diving into Data at Facebook · Lior Abraham et al. · 2013VLDB 2013; an in-memory system for real-time analysis of logs, with sampling.
- Kafka: a Distributed Messaging System for Log Processing · Jay Kreps, Neha Narkhede and Jun Rao · 2011The LinkedIn paper that introduced Kafka (NetDB workshop, 2011).
- Wikimedia analytics dumpsPublic, aggregated page-view counts for Wikipedia articles.
- UN Committee of Experts on Big Data and Data Science for Official Statistics
- US Census Bureau: differential privacy and the 2020 Census