Forty terabytes and a catalogue
A city museum in Kraków is digitising its collection: two million high-resolution scans, about 40 TB, which researchers worldwide will view online. It also has a catalogue — a relational database of objects, artists, loans and conservation records — and a small team of developers. The museum's director asks three questions that every cloud storage decision comes down to. Where should each kind of data live? How likely are we to lose any of it? And if something goes badly wrong — a region outage, a bug, a ransomware attack, a tired administrator typing a command into the wrong window — how much would we lose, and how long would we be closed?
The providers' marketing answers the second question with a number like "99.999999999% durability", eleven nines. This lesson explains what such a number means, computes it from a model, and shows which assumption in that model matters most. It then turns to the databases built on top of storage, the consistency they really offer, and backups. Lumenary CS's storage at scale (storage engines, partitioning, object stores), replication and consensus and databases lessons are the background; here we see how a cloud packages those ideas as services.
Block, file and object storage
Clouds offer three fundamentally different kinds of storage, distinguished by the interface they present.
Definition— Block, file and object storage
Block storage presents a virtual disk: an array of fixed-size blocks that one machine's operating system formats with a file system and reads and writes at any offset. File storage presents a shared file system — directories, files, permissions, locks — that many machines mount at once over a network protocol such as NFS or SMB. Object storage presents a flat namespace of buckets and keys, each holding an object written whole and read whole or by byte range, over an HTTP API.
| Block | File | Object | |
|---|---|---|---|
| Interface | a disk (/dev/nvme1n1) | a mounted directory (/mnt/shared) | HTTP: PUT, GET, DELETE a key |
| Shared by | one machine at a time (usually) | many machines | any number of clients, anywhere |
| Latency | well under a millisecond to a few ms | a few ms | tens of ms per request |
| Typical use | databases, a VM's system disk | shared home directories, render farms, legacy applications | media, backups, data lakes, static websites |
| Examples | Amazon EBS, Azure Managed Disks, Google Persistent Disk and Hyperdisk, Alibaba Cloud ESSD; Ceph RBD | Amazon EFS and FSx, Azure Files, Google Filestore, Alibaba Cloud NAS; CephFS, NFS servers | Amazon S3, Azure Blob Storage, Google Cloud Storage, Alibaba Cloud OSS; Ceph RGW |
The S3 API has become a de facto standard, accepted by many object stores — Alibaba Cloud OSS, Google Cloud Storage's interoperability mode, many regional providers and open-source servers such as Ceph's gateway — though Azure Blob Storage has its own API. Block storage is usually zonal: a disk lives in one zone and attaches only to machines there (some providers also offer regional disks, replicated to a second zone). Managed file systems and object stores are usually replicated across the zones of a region, and object stores across regions if you ask.
There is a fourth kind that is easy to overlook: instance storage, NVMe drives physically inside the host. It is the fastest storage there is, and it is ephemeral: when the VM is stopped or moved, or the host fails, the data is gone. It suits caches, scratch space and databases that replicate themselves across machines, never the only copy of anything.
Object stores also offer storage classes that trade price per gigabyte against the cost and delay of reading. As of 2026, archive tiers on the large clouds cost roughly a tenth or less of the standard tier per gigabyte-month, but charge for retrieval and for deleting an object before a minimum period (from 30 days to a year, depending on class and provider), and some take hours to bring an object back. Lifecycle rules move objects between classes by age automatically: the museum's scans might stay in the standard class while new, move to an infrequent-access class after a year, and keep a second copy in an archive class in another region.
What eleven nines means
Definition— Durability and availability
Durability is the probability that a stored object is not lost over a period, usually a year. Availability is the fraction of time the service answers requests successfully. A store can be unavailable for an hour without losing anything; it can be available while having quietly lost an object.
The two numbers differ enormously in practice. The large object stores state designed annual durabilities of eleven nines or more, while their availability targets and service-level agreements are typically three or four nines. Eleven nines means a probability of about of losing a given object in a year. For the museum's two million scans that is an expected objects a year: on average, one lost scan every 50,000 years — if the model behind the number is right.
Where does such a number come from? From a model of failure and repair. An object is stored as fragments — whole copies, or erasure-coded pieces — on different disks, and survives as long as no more than a certain number are lost at once. Disks fail at random; when one does, the system rebuilds its fragments elsewhere from the survivors. Data is lost only if enough further disks fail before the repairs finish. The block below computes this with a small Markov chain, for an illustrative annual disk failure rate of 2%.
import math
AFR = 0.02 # each disk fails with probability about 2% a year (an example)
lam = -math.log(1 - AFR) # the same, as a failure rate per disk per year
def mttdl_years(n, tolerate, repair_hours):
"""Mean time to data loss for an object stored as n fragments (copies or erasure-coded
pieces) on n different disks, which survives losing up to `tolerate` of them. Lost
fragments are rebuilt in parallel, each taking about repair_hours on average.
A Markov chain on f = fragments currently lost: f -> f+1 at rate (n - f) * lam,
f -> f-1 at rate f * mu. If tau_f is the expected time to first reach f+1 from f,
then tau_f = (1 + f * mu * tau_(f-1)) / ((n - f) * lam), and MTTDL is their sum."""
mu = 365 * 24 / repair_hours
tau, total = 0.0, 0.0
for f in range(tolerate + 1):
tau = (1 + f * mu * tau) / ((n - f) * lam)
total += tau
return total
print("scheme stores repair P(lose a given object in a year) nines")
for name, n, k in [("2 copies", 2, 1), ("3 copies", 3, 1), ("erasure code (6, 3)", 9, 6),
("erasure code (10, 4)", 14, 10)]:
for hours in [24, 1]:
p = -math.expm1(-1 / mttdl_years(n, n - k, hours)) # 1 - exp(-1/MTTDL), computed accurately
print(f"{name:20} {n / k:5.1f}x {hours:5d} h {p:22.1e} {-math.log10(p):20.1f}")
# The model assumes independent failures. If every fragment is in one zone, an event that
# destroys the zone (fire, flood, a faulty bulk deletion) caps durability, whatever the code.
for p_zone in [1e-4, 1e-6]:
print(f"all fragments in one zone, lost with probability {p_zone:.0e} a year: at most {-math.log10(p_zone):.0f} nines")Three copies with a day to repair give about 9.7 nines; repairing within an hour gives 12.5. A erasure code — ten data fragments plus four parity fragments, surviving any four losses — stores only 1.4 bytes per byte yet reaches 14.7 nines with day-long repairs. Two lessons follow. Fast repair is worth as much as extra copies: with copies, the loss probability falls roughly like , so cutting repair time 24-fold gains factors of 24. And erasure codes buy durability cheaply, which is why every large object store uses them for most data.
Replication and erasure coding
Replication is the simple answer: keep full copies on different disks, racks or zones. Reads can go to any copy, a lost copy is rebuilt by copying a survivor, and the logic is easy to verify. It costs bytes of disk per byte stored.
An erasure code with parameters splits data into fragments and computes extra parity fragments such that any of the reconstruct the original. Lumenary CS's storage-at-scale lesson met the Reed–Solomon code; the block below builds a small code of the same kind from scratch, using nothing but polynomials.
P = 257 # a prime: arithmetic modulo P is a field
def interpolate(points, x):
"""The value at x of the unique polynomial of degree < len(points) through `points`
(Lagrange interpolation, all arithmetic modulo P)."""
total = 0
for i, (xi, yi) in enumerate(points):
num, den = 1, 1
for j, (xj, _) in enumerate(points):
if j != i:
num = num * (x - xj) % P
den = den * (xi - xj) % P
total = (total + yi * num * pow(den, P - 2, P)) % P # division = multiply by inverse
return total
def encode(data, k, m):
"""Split data into k fragments and add m parity fragments: fragment i holds, for each
stripe of k symbols, the value at x = i + 1 of the polynomial through that stripe."""
data = data + bytes(-len(data) % k) # pad to whole stripes
stripes = [data[s:s + k] for s in range(0, len(data), k)]
return [[interpolate(list(zip(range(1, k + 1), st)), x) for st in stripes]
for x in range(1, k + m + 1)] # fragments 1..k are the data itself
def decode(fragments, k):
"""Rebuild the data from ANY k surviving fragments (None marks a lost one)."""
alive = [(x + 1, f) for x, f in enumerate(fragments) if f is not None][:k]
if len(alive) < k:
raise ValueError("too many fragments lost")
out = []
for s in range(len(alive[0][1])):
pts = [(x, f[s]) for x, f in alive]
out += [interpolate(pts, x) for x in range(1, k + 1)]
return bytes(out)
record = "Synthetic test record: archive scan 0042, digitised 2026-11-19".encode()
k, m = 6, 3
frags = encode(record, k, m)
print(f"{len(record)} bytes -> {k + m} fragments of {len(frags[0])} symbols "
f"(overhead {(k + m) / k:.2f}x, any {m} may be lost)")
print("data fragment 1 starts:", frags[0][:6], " parity fragment 7 starts:", frags[6][:6])
for lost in [(0, 3, 8), (1, 2, 4), (5, 6, 7)]:
damaged = [None if i in lost else f for i, f in enumerate(frags)]
print(f"lose fragments {[i + 1 for i in lost]}: rebuilt correctly?",
decode(damaged, k)[:len(record)] == record)
try:
decode([None, None, None, None] + frags[4:], k)
except ValueError as e:
print("lose 4 fragments:", e)The code is systematic: fragments 1 to 6 hold the data itself (interleaved, one symbol of each stripe), so a healthy read needs no decoding. Any three fragments can be lost — data, parity or a mixture — and the record comes back; a fourth loss is fatal. Production codes work in the field with elements rather than modulo 257, so that every symbol is exactly one byte (here a parity symbol may be 256), and use fast matrix methods rather than Lagrange's formula, but the principle is the same.
Erasure coding has costs that replication does not. Rebuilding one lost fragment means reading others, so a code moves ten times as much data across the network for a repair as replication would. Small objects are inefficient to split, so stores often replicate them and erasure-code large ones. And reads during a failure must decode. Engineers have attacked the repair cost directly: Azure's storage team described locally repairable codes, which add parity over small groups so that most repairs read only a few fragments, in a 2012 paper.
Placement completes the design. Storing fragments in three zones so that the loss of a whole zone leaves at least of them is what turns a code's durability into protection against the correlated failures Gideon described. Azure's redundancy options make this choice explicit — locally, zone- and geo-redundant storage — and the other providers offer similar choices through storage classes and multi-region buckets.
Managed databases
Most applications keep their important state not in raw storage but in a database, and in the cloud most teams rent it managed. A managed relational database — Amazon RDS and Aurora, Azure SQL Database and Azure Database for PostgreSQL and MySQL, Google Cloud SQL and AlloyDB, Alibaba Cloud ApsaraDB RDS and PolarDB, among others — runs PostgreSQL, MySQL, SQL Server or a compatible engine for you. What "managed" includes is worth listing precisely:
- Provided: installation and minor-version patching, automated backups with point-in-time recovery, a standby replica in another zone with automatic failover, read replicas, monitoring, encryption at rest, and storage that grows.
- Still yours: the schema and indexes, query performance, connection management, access control (who can connect, with what privileges), choosing backup retention and testing restores, major-version upgrades (scheduled by you, tested by you), and the bill.
Several of these services separate compute from storage. Amazon Aurora, for example, sends only its redo log to a shared storage layer that keeps six copies across three zones and accepts a write once four have it, as its SIGMOD 2017 paper explains; Azure SQL Hyperscale, PolarDB, AlloyDB and the open-source Neon follow related designs. Failover and new replicas become faster because the storage is already shared.
Beyond one primary per database, two other families exist. Distributed SQL databases partition and replicate tables across many machines and zones, with consensus for each partition and transactions across them: Google's Spanner and open-source or source-available systems such as CockroachDB, YugabyteDB, TiDB and OceanBase. NoSQL services trade the relational model for predictable scaling on simpler access patterns: key-value and document stores such as Amazon DynamoDB, Azure Cosmos DB, Google Firestore and Bigtable, and Alibaba Cloud Tablestore, with Apache Cassandra, ScyllaDB and MongoDB among the self-run options.
A managed database is a trade: less operational work, more dependence on one provider's product, its limits and its price. Engines that are open source (PostgreSQL, MySQL) keep the option of moving; proprietary APIs make moving a rewrite. Neither choice is wrong, but it should be a choice.
Consistency in practice
Lumenary CS's replication-and-consensus lesson defined linearizability and eventual consistency. In the cloud the question becomes concrete: if I write and then read, will I see my write?
Within one primary database, yes. The trouble starts with read replicas, which most managed relational databases update asynchronously: a write commits on the primary and reaches each replica some milliseconds — occasionally seconds — later. A read sent to a replica in that window returns the old value. The block uses an illustrative lag distribution: usually about 20 ms, but 5% of the time the replica is busy and lags about a second.
import math
# Replication lag of an asynchronous read replica, as a mixture (illustrative numbers):
# usually it applies a write within ~20 ms, but 5% of the time it is busy and lags ~1 s.
def p_stale(delay_s):
"""Probability that a read sent delay_s after a write still sees the old value."""
return 0.95 * math.exp(-delay_s / 0.020) + 0.05 * math.exp(-delay_s / 1.0)
print("read issued after the write P(stale read from replica)")
for d in [0.005, 0.05, 0.2, 1, 5]:
print(f"{d * 1000:10.0f} ms {p_stale(d):27.2%}")The usual remedy is a session guarantee rather than strong consistency everywhere: read-your-writes (a client always sees its own writes) and monotonic reads (it never sees time go backwards). Applications get them by sending a user's reads to the primary for a short while after that user writes, or by carrying a log position from the write and reading only from a replica that has applied it. Services expose such choices explicitly. Azure Cosmos DB offers five consistency levels, from strong through bounded staleness, session and consistent prefix to eventual; DynamoDB's reads are eventually consistent by default with a strongly consistent option within a region; Spanner provides externally consistent transactions across regions using tightly synchronised clocks; and the major object stores, after years of eventual consistency for some operations, now give strong read-after-write consistency for objects (Amazon S3 since December 2020).
Across regions the speed of light returns. A synchronous write to another continent costs at least one intercontinental round trip, the networking lesson's 100 ms or more, so most multi-region designs replicate asynchronously and accept that the newest writes can be lost if a region fails — which is exactly the RPO of the next section.
Backups and disaster recovery
Definition— RPO and RTO
The recovery point objective (RPO) is the most data, measured in time, that you are prepared to lose in a disaster: an RPO of 5 minutes means the restored system may lack at most the last 5 minutes of changes. The recovery time objective (RTO) is the longest you are prepared to be out of service before the system works again.
Both are business decisions, and both cost money: a smaller RPO needs more frequent copies or continuous replication; a smaller RTO needs more of the system already running somewhere else. The block compares four designs for the museum's 2 TB catalogue database, with illustrative throughput figures.
# Recovery for a 2 TB catalogue database, four designs (illustrative figures).
DB_GB = 2000
RESTORE_GB_PER_H = 1800 # copying a snapshot back from object storage, about 0.5 GB/s
WAL_GB_PER_H = 5 # change log the database writes per hour
REPLAY_GB_PER_H = 50 # how fast a restored copy can replay that log
PROVISION_H = 0.25 # starting a new database server
def restore_hours(snapshot_every_h):
worst_log = snapshot_every_h * WAL_GB_PER_H # log written since the last snapshot
return PROVISION_H + DB_GB / RESTORE_GB_PER_H + worst_log / REPLAY_GB_PER_H
designs = [
# name, worst-case RPO in hours, worst-case RTO in hours
("nightly snapshot only", 24, PROVISION_H + DB_GB / RESTORE_GB_PER_H),
("nightly snapshot + log archived every 5 min", 5 / 60, restore_hours(24)),
("4-hourly snapshot + log archived every 5 min", 5 / 60, restore_hours(4)),
("warm standby in another region (async, ~2 s lag)", 2 / 3600, 10 / 60),
]
print("design worst RPO worst RTO")
for name, rpo, rto in designs:
rpo_text = f"{rpo:.0f} h" if rpo >= 1 else (f"{rpo * 60:.0f} min" if rpo * 60 >= 1 else f"{rpo * 3600:.0f} s")
print(f"{name:50} {rpo_text:>10} {rto * 60:9.0f} min")Archiving the database's change log every five minutes, as PostgreSQL's point-in-time recovery and every managed service do, cuts the worst-case RPO from a day to five minutes — but lengthens the worst-case RTO from 82 to 226 minutes, because a restore just before the next nightly snapshot must replay almost a day of log. Snapshotting every four hours brings it back to 106 minutes. Only a standby already running in another region gets both down to seconds and minutes, at the price of a second database running all the time. Industry guides name a ladder of such designs — backup and restore, a "pilot light" of minimal standby infrastructure, warm standby, and active–active across regions — each costing more and recovering faster.
Five habits separate backups that work from backups that exist:
- Replication is not a backup. A mistaken
DELETE, a buggy migration or ransomware is copied to every replica within seconds. Backups are copies from the past. - Keep copies apart: the old 3-2-1 rule — three copies, on two kinds of media, one off-site — translates to copies in another region and, ideally, another account with separate credentials.
- Make some copies immutable. Object-lock or write-once retention settings stop anyone, including an attacker with stolen administrator credentials, from deleting or altering a backup before its retention ends.
- Protect the keys. Encrypted backups are only as recoverable as the keys that decrypt them; a key deleted with the account takes every backup with it.
- Test restores, regularly and timed. A backup that has never been restored is a hope, not a plan, and the measured restore time is your real RTO.
Backups of personal data also carry obligations: data-protection laws such as the GDPR expect retention periods to be justified, and deletion requests to reach backups eventually, typically by letting backups expire on a defined schedule rather than editing them.
Common misconceptions
Misconception
"Eleven nines of durability means my data is safe." It means the provider's storage is very unlikely to lose an object through hardware failure. It says nothing about you or your software deleting or overwriting it, about an attacker with your credentials, or about losing access to your account. Those are covered by versioning, immutable backups and access control — your side of the shared-responsibility model.
Misconception
"Durability and availability are the same thing." A store can be unreachable without losing data, and durability figures (eleven nines) are far higher than availability figures (three or four nines). Design for the store being unavailable sometimes.
Misconception
"A multi-zone database is a backup." A standby in another zone protects against losing a zone, not against a bad write, which it faithfully replicates. You need both: replicas for availability, backups for recovery.
Misconception
"Data on the VM's disk is stored durably." Network block volumes are replicated within their zone; local instance storage disappears with the VM or the host. Know which kind each path on your machine is.
Exercises
1. Storage and managed databases
Warm-upHumanA film-restoration studio in Mumbai needs storage for three things: (1) the data directory of the PostgreSQL server that tracks its projects; (2) a directory of shared scripts and reference frames that 40 rendering machines mount at the same time; (3) 300 TB of finished, rarely changed video masters that partners download over HTTPS. Which storage kinds fit best, in that order?
2. Storage and managed databases
CoreHumanA national library stores 500 million objects in an object store with a designed annual durability of eleven nines (a probability of of losing a given object in a year, independently). On average, how many years pass between losses of a single object?
3. Storage and managed databases
Warm-upHumanA storage cluster protects data with a erasure code: every object is split into 12 data fragments, 4 parity fragments are added, and any 12 of the 16 suffice to rebuild it. How many TB of raw disk does it need to store 600 TB of data (ignoring metadata)?
4. Storage and managed databases
CoreHumanA storage service promises 99.9% availability per month. In a 30-day month, how many seconds of unavailability can occur before the promise is broken?
5. Storage and managed databases
CoreHumanA clinic-booking service in Bogotá takes a database snapshot every 6 hours (at 00:00, 06:00, 12:00 and 18:00) and keeps no change log. At 17:40 a faulty migration corrupts the database. The team restores the 12:00 snapshot and the service is working again at 19:10. What recovery point and recovery time did they actually achieve?
6. Storage and managed databases
CoreHumanA company's managed database has a synchronous standby in a second zone and an asynchronous replica in another region. An engineer accidentally runs DELETE FROM orders; without a WHERE clause on the primary. Which of these will let the company get its orders back?
7. Storage and managed databases
StretchHumanA language-learning app in Hanoi sends all writes to its database's primary and all reads to two asynchronous read replicas, to spread the load. Users report that after they change their daily goal, the settings page sometimes still shows the old goal, and shows the new one if they reload a few seconds later. Which change fixes this with the least extra load on the primary?
8. Storage and managed databases
CoreComputationalThe simplest erasure code adds one parity block: the byte-by-byte XOR of equal-length data blocks. It survives the loss of any one block. Write xor_parity(blocks), which returns that parity as bytes, and rebuild(blocks, parity), which receives the data blocks with exactly one replaced by None and returns a new list with the lost block recomputed. Do not modify the list you are given.
The starter's functions do not compute anything yet.
Further reading and resources
- Amazon S3 user guide
- Azure Storage redundancy
- Google Cloud Storage documentation
- Alibaba Cloud Object Storage Service (OSS)
- CephOpen-source software providing block, file and object storage on your own hardware.
- Erasure Coding in Windows Azure Storage · Cheng Huang and colleagues · 2012
- Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases · Alexandre Verbitski and colleagues · 2017
- Spanner: Google's Globally-Distributed Database · James C. Corbett and colleagues · 2012
- Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service · Mostafa Elhemali and colleagues · 2022
- Consistency levels in Azure Cosmos DB
- PostgreSQL: continuous archiving and point-in-time recovery