Five jobs, one plan
A public-transport authority in Bogotá processes a day of smart-card taps — a few hundred million rows — every night. In the Hadoop era its pipeline was five MapReduce jobs in a row: clean the taps, join them with the stop list, build journeys, aggregate by route and hour, rank the busiest stops. Each job read its input from the distributed file system and wrote its full output back, replicated three times, for the next job to read. Most of the night went on writing and re-reading intermediate files that nobody wanted.
Batch processing and MapReduce showed why MapReduce is so robust: deterministic tasks, re-run on failure. Its weakness is that it sees one map and one reduce at a time. A dataflow engine sees the whole computation as a graph before running any of it, so it can fuse steps, keep intermediate data in memory or on local disk, choose join strategies, and recompute only what is lost. Apache Spark is the most widely used such engine for batch work as of 2026, and the same ideas run through Apache Flink, Apache Beam, Dask, Ray Data, and the distributed SQL engines of Analytical databases.
This lesson builds a tiny Spark in Python to see how laziness, stages and lineage work; reads a real query plan; simulates the two problems that cost Spark users most — shuffles and skew; and ends with the question every team should ask first: do we need a cluster at all?
RDDs, lineage and laziness
Batch processing and MapReduce ended with Spark's founding idea, from Matei Zaharia's work at UC Berkeley's AMPLab (the 2012 paper is linked below). Here it is precisely.
Definition— Resilient distributed dataset (RDD)
An RDD is an immutable collection of records divided into partitions, which may live on different machines. It is defined not by its contents but by its lineage: the parent RDDs it came from and the function that turns their partitions into its own. Operations are of two kinds. Transformations (map, filter, join, reduceByKey, …) return a new RDD and compute nothing. Actions (count, collect, take, writing a file) ask for a result, and only then does Spark run the work needed to produce it.
Laziness is not a trick to save typing. Because nothing runs until an action, Spark sees the whole chain before choosing how to execute it. And because every partition can be rebuilt from its lineage, Spark does not need to replicate intermediate results to survive failures: if a machine dies, it recomputes the lost partitions from their parents, in parallel, on other machines.
A Spark application has one driver, the process that runs your program, builds the plan and schedules work, and many executors, processes on worker machines that run tasks and hold data. A cluster manager — Kubernetes, Hadoop YARN, Spark's own standalone manager, or a cloud service — starts the executors. A task is one stage's work on one partition, so the number of partitions sets the parallelism.
Here is word-count's cousin in the RDD API: total ride minutes per city from a CSV file (not run here; it needs Spark).
# Not run here: PySpark, RDD API.
lines = spark.sparkContext.textFile("s3://example-bucket/rides/2026-03-14/*.csv")
minutes = (lines.map(lambda line: line.split(","))
.filter(lambda f: len(f) == 2 and f[1].isdigit())
.map(lambda f: (f[0], int(f[1])))
.reduceByKey(lambda a, b: a + b)) # still nothing has run
minutes.take(5) # an action: now the job runsA Spark you can read
The block below implements an RDD in about fifty lines. Partitions are lists; a narrow transformation wraps its parent's iterator for the same partition; reduce_by_key is a shuffle, written as a dict of lists. Each RDD knows its parent, so it can print its stages and recompute a lost partition. A counter records how often the user's parse function runs, which makes laziness, caching and recovery visible.
import random
import zlib
from collections import Counter
calls = Counter() # how often each user function has run
def part_of(key, n):
"""The partition a key goes to: a stable hash, so every worker agrees."""
return zlib.crc32(str(key).encode()) % n
class RDD:
"""A toy resilient distributed dataset: n partitions, each computed lazily, on demand,
from its parent. Nothing runs until an action asks for data."""
def __init__(self, n, parent, compute, wide=False, label=""):
self.n, self.parent, self.compute, self.wide, self.label = n, parent, compute, wide, label
self.cached = None # partition -> list, once cache() is called
@staticmethod
def parallelize(data, n):
return RDD(n, None, lambda i: iter(data[i::n]), label="parallelize")
# Narrow: partition i of the result needs only partition i of the parent.
def map(self, f):
return RDD(self.n, self, lambda i: map(f, self.iterator(i)), label="map")
def filter(self, f):
return RDD(self.n, self, lambda i: filter(f, self.iterator(i)), label="filter")
# Wide: partition i of the result needs a piece of EVERY parent partition, so a shuffle.
def reduce_by_key(self, f, n):
buckets = {} # the shuffle files, kept once written
def compute(i):
if not buckets: # map side: each parent partition writes n buckets
for j in range(self.n):
local = {}
for k, v in self.iterator(j): # combine locally before sending
local[k] = f(local[k], v) if k in local else v
for k, v in local.items():
buckets.setdefault(part_of(k, n), []).append((k, v))
total = {}
for k, v in buckets.get(i, []): # reduce side: fetch bucket i from everyone
total[k] = f(total[k], v) if k in total else v
return iter(sorted(total.items()))
return RDD(n, self, compute, wide=True, label="reduceByKey")
def cache(self):
self.cached = {}
return self
def iterator(self, i):
if self.cached is None:
return self.compute(i)
if i not in self.cached:
self.cached[i] = list(self.compute(i))
return iter(self.cached[i])
# Actions: only these run anything.
def collect(self):
return [x for i in range(self.n) for x in self.iterator(i)]
def stages(self):
"""Cut the lineage at every wide dependency; each stage is a chain of narrow steps."""
chain, node = [], self
while node is not None:
chain.append(node.label)
node, wide = node.parent, node.wide
if wide:
chain.append("|")
return " -> ".join(reversed(chain)).split(" -> | -> ")
# Trip logs from a bike-share network: "city,minutes". A few lines are corrupt.
rng = random.Random(4)
cities = ["Accra", "Bogota", "Chennai", "Krakow", "Osaka", "Tunis"]
lines = [f"{rng.choice(cities)},{rng.randint(3, 60)}" for _ in range(12_000)]
lines[100], lines[7_000] = "Osaka,", "###"
def parse(line):
calls["parse"] += 1
city, _, minutes = line.partition(",")
return city, int(minutes) if minutes.isdigit() else None
trips = (RDD.parallelize(lines, 6) # 6 input partitions
.map(parse)
.filter(lambda t: t[1] is not None))
hours_by_city = (trips.reduce_by_key(lambda a, b: a + b, 3) # 3 partitions after the shuffle
.map(lambda kv: (kv[0], kv[1] // 60)))
trips_by_city = trips.map(lambda t: (t[0], 1)).reduce_by_key(lambda a, b: a + b, 3)
print("after defining two queries, parse has run", calls["parse"], "times")
print("stages:", hours_by_city.stages())
print("hours by city:", dict(hours_by_city.collect()))
print("parse calls after the first action: ", calls["parse"])
trips_by_city.collect()
print("parse calls after the second action:", calls["parse"], "(the input was parsed again)")
calls.clear()
trips.cache() # keep trips' partitions once computed
minutes = dict(trips.reduce_by_key(lambda a, b: a + b, 3).collect())
counts = dict(trips.map(lambda t: (t[0], 1)).reduce_by_key(lambda a, b: a + b, 3).collect())
print("\nwith trips cached, both actions together parse", calls["parse"], "lines")
del trips.cached[2] # an executor dies, losing cached partition 2
calls.clear()
assert dict(trips.reduce_by_key(lambda a, b: a + b, 3).collect()) == minutes
print("after losing partition 2, lineage recomputes it alone:", calls["parse"], "lines parsed")Four behaviours of real Spark appear in miniature:
- Laziness. Defining both queries parsed nothing. The first
collectparsed all 12,000 lines. - Stages. The plan splits in two at the shuffle:
parallelize -> map -> filterruns as one pipelined pass over each input partition, andreduceByKey -> mapruns over each of the three output partitions. Within a stage, a record flows throughparseand the filter without any intermediate list being built — Python's lazymapandfilterplay the part of Spark's pipelining. - Recomputation without a cache. The second query shares
tripswith the first, but nothing kepttrips, so all 12,000 lines were parsed again: 24,000 calls in all. Withtrips.cache(), two queries cost one parse. - Lineage recovery. When cached partition 2 was "lost", only its 2,000 lines were parsed again, from the lineage, and the answer was unchanged.
The toy also keeps each shuffle's output once written, as Spark keeps shuffle files on executors' local disks; a later job that needs the same shuffle shows it in Spark's web interface as a skipped stage.
Narrow and wide: where the stages break
Definition— Narrow and wide dependencies
A transformation has a narrow dependency if each partition of its output depends on at most one partition of each parent (or a fixed, small set of them): map, filter, flatMap, union, and coalesce to fewer partitions. It has a wide dependency if an output partition may need data from every parent partition: groupByKey, reduceByKey, distinct, repartition, sorting, and joins whose inputs are not already partitioned the same way. A wide dependency requires a shuffle, and the scheduler cuts the job into stages at every shuffle.
A shuffle is the expensive moment of almost every distributed job. Each task of the earlier stage partitions its output by key and writes it to local disk (the shuffle write); each task of the later stage fetches its piece from every earlier task over the network (the shuffle read), then sorts or hashes it. With map tasks and reduce partitions there are pieces, so the network carries roughly the whole dataset, disks are written and read, and stage 2 cannot start until the slowest task of stage 1 has finished. If an executor holding shuffle files dies, the tasks that wrote them are re-run from lineage; Spark can also run an external shuffle service so that shuffle files outlive executors.
Two consequences shape how people write Spark. First, cutting the data down before a shuffle pays twice: filter early, select only the columns you need, and let map-side combining (the combiner of MapReduce, which reduceByKey does and groupByKey does not) shrink each key to one record per map task. Second, a shuffle can sometimes be avoided entirely: a broadcast join copies a small table to every executor instead of shuffling the big one — Analytical databases proved when that moves fewer bytes — and two tables already bucketed by the join key into the same number of buckets can be joined partition by partition.
DataFrames and the query plan
Most Spark code written today does not use RDDs directly. The DataFrame API (2015) gives each dataset a schema of named, typed columns, and expresses work as relational operations — select, filter, join, group, aggregate — in Python, Scala, Java, R or SQL. Because Spark now knows what each operation means, not just that it is some function, its optimiser, Catalyst, can rewrite the query as a database would: push filters down to the data source, read only the needed columns, fold constants, reorder and choose join algorithms. A second layer generates compact code for whole stages and stores rows in a compact binary format rather than as language objects.
The job below is the bike-share report as DataFrame code (not run here: it needs a Spark cluster and real storage).
# Not run here: PySpark, DataFrame API.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("long-member-rides").getOrCreate()
rides = spark.read.parquet("s3://example-bucket/rides/") # reads only metadata for now
stations = spark.read.parquet("s3://example-bucket/stations/") # a few thousand rows
report = (rides
.where(F.col("member") & (F.col("minutes") > 30))
.join(F.broadcast(stations), "station_id") # hint: copy the small side
.groupBy("city", "station_name")
.agg(F.count("*").alias("rides"), F.avg("minutes").alias("mean_minutes")))
report.explain() # print the plan; runs nothing
report.write.mode("overwrite").partitionBy("city").parquet("s3://example-bucket/reports/long-rides/")explain() prints the physical plan. Here is an abbreviated and simplified version of the kind of plan Spark prints for this query (the exact text, identifiers and operators vary between versions and settings):
AdaptiveSparkPlan
+- HashAggregate(keys=[city, station_name], functions=[count(1), avg(minutes)])
+- Exchange hashpartitioning(city, station_name, 200)
+- HashAggregate(keys=[city, station_name], functions=[partial_count(1), partial_avg(minutes)])
+- Project [minutes, city, station_name]
+- BroadcastHashJoin [station_id], [station_id], Inner, BuildRight
:- Filter (member AND (minutes > 30.0))
: +- FileScan parquet [station_id, minutes, member]
: PushedFilters: [EqualTo(member,true), GreaterThan(minutes,30.0)]
+- BroadcastExchange
+- FileScan parquet [station_id, city, station_name]Read it from the bottom up, as data flows:
FileScanlists three columns, not every column of the rides table: column pruning.PushedFiltersare handed to the Parquet reader, which can skip whole row groups using their statistics, as in Data models and file formats. (Spark still re-applies the filter row by row above the scan, since row-group skipping is only coarse.)BroadcastExchangeandBroadcastHashJoin: the stations table is sent to every executor and joined there; the rides are not shuffled for the join.partial_countandpartial_avgrun before the exchange: map-side combining. An average is carried as a running sum and count, which can be merged; this is whyavgcan be partially aggregated but a median cannot.Exchange hashpartitioning(…, 200)is the shuffle, the stage boundary. The 200 is Spark's default number of shuffle partitions (the settingspark.sql.shuffle.partitions).AdaptiveSparkPlan: with adaptive query execution (AQE), on by default since Spark 3.2, Spark re-plans at each stage boundary using the real sizes of shuffle output. It can merge many small shuffle partitions into fewer, switch a sort-merge join to a broadcast join when one side turns out small, and split skewed partitions of a join.
Spark's own trajectory continues: Spark Connect (introduced in 3.4) separates thin clients from the cluster over a remote protocol, and Spark 4.0 (2025) turned on ANSI SQL behaviour by default, so that, for example, an invalid cast raises an error instead of silently producing a null. Several projects replace Spark's execution layer with native, vectorised C++ or Rust code while keeping its API: Databricks' proprietary Photon engine, and the open-source Apache Gluten (with Meta's Velox library or ClickHouse as backends) and Apache DataFusion Comet.
Partitions, shuffles and skew
How many partitions should a shuffle have? Each task has a fixed overhead — scheduling, starting, fetching its pieces — of the order of tens to hundreds of milliseconds, and each must fit its share of the data in the executor's memory or spill it to disk.
Even a well-chosen count fails when the data are skewed: hash partitioning sends every record with one key to one partition, so if a key has 30% of the records, one task does about 30% of the stage's work while the others finish and wait. A stage takes as long as its slowest task. Skew is the most common reason a Spark job that ran in ten minutes yesterday runs for two hours today: a viral product, a default value such as an empty string or null shared by millions of rows, a bot that generated half the traffic.
Batch processing and MapReduce measured skew in a counting job, where a combiner removes most of it, and spread hot keys with a random salt: a second component added to the shuffle key of hot keys, so that their records spread over several partitions, followed by a second, much smaller aggregation that removes the salt and combines the partial results. The block below takes a harder query, distinct users per product, which has no compact combiner — every view of a product by the same user must meet so that the user is counted once — and where the choice of salt decides whether the answer is right. One product has gone viral.
import random
import zlib
from collections import Counter, defaultdict
def h(x):
return zlib.crc32(str(x).encode()) # a stable hash that every worker computes alike
P = 16 # reduce partitions
rng = random.Random(7)
# Product views: (product, user). One product went viral and gets 30% of all views.
events = []
for _ in range(400_000):
product = "P0001" if rng.random() < 0.30 else f"P{rng.randrange(2, 5_000):04d}"
events.append((product, f"u{rng.randrange(150_000)}"))
def shuffle(rows, key):
"""Send each row to partition hash(key) mod P: a shuffle is a dict of lists."""
parts = defaultdict(list)
for row in rows:
parts[h(key(row)) % P].append(row)
return parts
def report(name, parts):
sizes = [len(parts[p]) for p in range(P)]
mean = sum(sizes) / P
print(f"{name:34s} busiest {max(sizes):7,} rows, mean {mean:9,.0f}, ratio {max(sizes) / mean:4.1f}")
# Query: distinct users per product. Every user of a product must meet in one place.
plain = shuffle(events, key=lambda e: e[0])
report("plain shuffle by product", plain)
truth = {}
for rows in plain.values():
users = defaultdict(set)
for product, user in rows:
users[product].add(user)
truth.update({p: len(u) for p, u in users.items()})
# Find hot keys from a 1% sample, as an engine or an engineer would.
sample = Counter(p for p, _ in rng.sample(events, 4_000))
hot = {p for p, c in sample.items() if c / 4_000 > 0.01}
print("hot keys found in the sample:", hot)
def salted_distinct(events, salt_of, S=16):
# Stage 1: the key becomes (product, salt) for hot products, spreading them over S partitions.
salted = [((p, salt_of(u, S) if p in hot else 0), u) for p, u in events]
parts = shuffle(salted, key=lambda e: e[0])
partial = []
for rows in parts.values():
users = defaultdict(set)
for key, user in rows:
users[key].add(user)
partial += [(key[0], len(u)) for key, u in users.items()]
# Stage 2: drop the salt and add the partial counts: a tiny shuffle, at most S rows per hot key.
final = Counter()
for product, n in partial:
final[product] += n
return parts, len(partial), final
parts, sent, by_user_hash = salted_distinct(events, salt_of=lambda u, S: h(u) % S)
report("salted by hash(user) mod 16", parts)
print(f"stage 2 shuffles {sent:,} partial counts; correct: {dict(by_user_hash) == truth}")
parts, _, by_random = salted_distinct(events, salt_of=lambda u, S: rng.randrange(S))
report("salted at random", parts)
print(f"correct: {dict(by_random) == truth}; viral product: true {truth['P0001']:,} users, "
f"random salt says {by_random['P0001']:,}")Without salting, the busiest of the 16 partitions receives 137,044 rows against a mean of 25,000 — 5.5 times the average, so the stage takes about 5.5 times as long as it should. Salting the one hot key (found from a 1% sample) brings the busiest partition down to 32,718, 1.3 times the mean, at the price of a second shuffle of only 5,014 partial counts.
The last two lines are the important ones. Salting at random balances the load just as well and gives the wrong answer: it reports 116,936 distinct users for the viral product instead of 82,454, because a user who viewed it several times lands in several salts and is counted once in each. Salting by a hash of the user puts all of one user's views in the same salt, so the partial counts are over disjoint sets of users and can be added.
In PySpark the distinct-count salt is two aggregations (not run here):
# Not run here: PySpark. Salt only the hot product; pmod keeps the salt non-negative.
S = 16
users_per_product = (views
.withColumn("salt", F.when(F.col("product") == "P0001", F.pmod(F.hash("user"), F.lit(S)))
.otherwise(F.lit(0)))
.groupBy("product", "salt").agg(F.countDistinct("user").alias("users"))
.groupBy("product").agg(F.sum("users").alias("users")))Before salting by hand, check the cheaper remedies. Filter out the junk key (often null or an empty string) if it is not meant to be in the result at all. Use an aggregation that combines on the map side. Let adaptive execution split skewed join partitions, which it does automatically when spark.sql.adaptive.skewJoin.enabled is on (the default). Broadcast the small side of a join if it fits. And remember that repartition(n, "product") does not cure skew: it hashes by the same key and puts the hot key in one partition again.
Caching, and when not to
df.cache() (or persist() with an explicit storage level) asks Spark to keep a dataset's partitions on the executors — in memory, spilling to local disk if memory runs short — the first time an action computes them, so later actions reuse them instead of recomputing the lineage. It is itself lazy: nothing is stored until an action runs. unpersist() releases it.
Example— Is this worth caching?
A data scientist in Nairobi cleans 400 GB of raw mobile-money transaction logs (synthetic, in this example) into a 60 GB DataFrame of valid transactions — an expensive step involving parsing and a join — and then computes five different reports from it, each an action.
Without a cache, every report re-reads and re-cleans the 400 GB: five full passes. With clean.cache() before the first report, the first action pays for cleaning plus storing 60 GB in executor memory and disk, and the other four read the cached copy. If the cluster has, say, 20 executors with 8 GB of storage memory each — 160 GB in all — the 60 GB fits, and caching is a clear win.
Now change one thing: the analyst caches the raw 400 GB instead of the cleaned result. It does not fit in 160 GB; partitions spill to disk or are evicted and recomputed, memory pressure slows every other task, and the job may well run slower than with no cache at all. Cache the smallest dataset that is reused, after the filters, and only when it is reused.
Two related tools are worth knowing. Checkpointing writes a dataset to reliable storage and cuts its lineage, which matters for iterative algorithms whose lineage would otherwise grow with every iteration. And for data reused across jobs, days or teams, a cache is the wrong tool: write a table (Parquet, or a table format from Lakehouses and table formats) instead.
When one machine is enough
In 2015 Frank McSherry, Michael Isard and Derek Murray asked of several published distributed graph-processing systems: how many cores does each need before it beats a competent single-threaded program on a laptop? They called that number the COST (the configuration that outperforms a single thread), and for several systems it was in the hundreds of cores, or unbounded — some never beat the laptop at all. Distribution has a price: serialisation, network transfer, coordination, stragglers and failures. It pays off only when the work is large enough to amortise that price.
Hardware moved the threshold a long way. A rented machine with a hundred or more cores, a terabyte or more of memory and fast local SSDs is ordinary in 2026, and embedded columnar engines — DuckDB, Polars, Apache DataFusion, pyarrow's compute functions — process hundreds of millions of rows per second on such a machine, spilling to disk for data larger than memory. The block below measures one: a filter, group-by and sort over five million rows, on a single core.
import time
import numpy as np
import pyarrow as pa
import pyarrow.compute as pc
pa.set_cpu_count(1) # one core, as in the browser, to be fair to the cluster
rng = np.random.default_rng(3)
n = 5_000_000 # five million bike trips, as one Arrow table in this process
trips = pa.table({
"station": rng.integers(0, 2_000, n, dtype=np.int32),
"minutes": rng.gamma(2.0, 9.0, n).astype(np.float32),
"member": rng.random(n) < 0.6,
})
bytes_per_row = trips.nbytes / n
print(f"{n:,} rows, {trips.nbytes / 1e6:.0f} MB in memory")
t0 = time.perf_counter()
long_member_trips = trips.filter(pc.and_(trips["member"], pc.greater(trips["minutes"], 30)))
per_station = (long_member_trips.group_by("station")
.aggregate([("minutes", "count"), ("minutes", "mean")])
.sort_by([("minutes_count", "descending")]))
seconds = time.perf_counter() - t0
for row in per_station.slice(0, 3).to_pylist():
print(f"station {row['station']:4d}: {row['minutes_count']} long member trips, "
f"mean {row['minutes_mean']:.1f} min")
rate = n / seconds
print(f"\nfilter, group by and sort: {seconds * 1000:.0f} ms, {rate / 1e6:.0f} million rows per second")
# Extrapolate to a billion rows, adding the time to read them at 2 GB/s (a fast local SSD).
rows = 1e9
gb = rows * bytes_per_row / 1e9
print(f"a billion rows: {gb:.1f} GB; compute about {rows / rate:.0f} s on one core, "
f"plus about {gb / 2:.0f} s to read")On the machine that checked this lesson, the query took about 20 milliseconds — over 200 million input rows per second on one core — so a billion such rows would need a few seconds of compute and a few more to read 8 GB. Your browser runs Python compiled to WebAssembly and will be several times slower; the conclusion survives. A Spark cluster, by contrast, spends tens of seconds to minutes starting executors (less on serverless offerings, which keep warm capacity), and every stage pays scheduling and shuffle overheads before it does any useful work.
So the honest decision rule, before any benchmark, is about size, shape and people:
- Up to hundreds of gigabytes, especially if the job is a SQL-shaped transformation run by a small team, start with one machine and an embedded engine. It is cheaper, faster to develop and test (the same code runs on a laptop), and there is no cluster to operate.
- Many terabytes and up, joins of two huge tables, or a single job that must finish faster than one machine can read its input, need distribution: Spark, a distributed SQL engine such as Trino, a cloud warehouse, or Flink's batch mode. So does processing that is naturally parallel in Python — feature extraction, model inference, image decoding — where Dask and Ray are common choices.
- The organisation's platform often decides. If every other pipeline runs on Spark, with shared monitoring, access control and on-call, one more Spark job may cost less in people than a new tool, even if a single machine would be faster.
Spark itself is open source (Apache 2.0). It runs on Kubernetes or YARN clusters you operate, or as a managed service: Databricks (on several clouds), Amazon EMR, Google Cloud Dataproc, Microsoft Fabric and Azure Synapse, Alibaba Cloud EMR, and others. Be plain about the operational burden either way. Someone must size executors and memory, choose partition counts, read plans when a job slows down, upgrade versions, manage Python dependencies on every node, and keep idle clusters from running up bills; managed services remove some of this work and charge for it. Spot or preemptible machines cut compute prices substantially for batch jobs, and lineage-based recovery is what makes losing them tolerable.
Common misconceptions
Misconception
"Spark is in-memory, so it never touches the disk." Shuffles always write to local disk, partitions that do not fit in memory spill, and input usually comes from files. Spark avoids writing intermediate results to replicated storage between steps, which is a different and more modest claim.
Misconception
"More partitions means more parallelism, so more is faster." Beyond the number of cores and a sensible size per task, more partitions add scheduling overhead and multiply shuffle blocks. Too few partitions spill and leave cores idle. Size partitions by bytes, and let adaptive execution coalesce them.
Misconception
"Caching everything makes jobs faster." A cache costs memory that tasks need for execution, and data used once gains nothing from it. Cache a dataset only when several actions reuse it and it fits; release it when done.
Misconception
"collect() is a harmless way to look at the result." It copies the whole dataset to the driver, one process on one machine; on a large result it fails with an out-of-memory error or takes the driver down. Use show(), take(n) or limit(n) to look, and write large results to storage.
Misconception
"If a job is slow, add machines." A skewed stage is as slow as its busiest task, and adding machines does not split one key's records. Look at the per-task times in the Spark UI first: if one task takes far longer than the median, fix the skew, not the cluster size.
Exercises
1. Distributed processing with Spark
Warm-upHumanWhich of these transformations have a wide dependency, and so need a shuffle? Assume no input is already partitioned by the relevant key. Select all that apply.
2. Distributed processing with Spark
Warm-upHumanA job is a straight chain of operations:
read → flatMap → filter → reduceByKey → map → distinct → map → repartition → write
Using the lesson's rule (a new stage starts at every wide dependency), how many stages does it have?
3. Distributed processing with Spark
CoreHumanA shuffle sends 10,000,000 records to 40 partitions by hash of the key. One key accounts for 22% of the records; the remaining records have so many different keys that they spread evenly over all 40 partitions. How many records does the busiest partition receive?
4. Distributed processing with Spark
CoreHumanIn the lesson's physical plan, a HashAggregate with partial_count(1), partial_avg(minutes) sits below the Exchange hashpartitioning(...), and another HashAggregate with count(1), avg(minutes) sits above it. What does this tell you?
5. Distributed processing with Spark
CoreComputationalWrite plan_stages(ops), the core of a stage planner for a straight chain of operations. It returns the list of stages, each a list of operation names in order, starting a new stage at every operation in WIDE. The first operation always starts the first stage, and an empty job has no stages.
For example, plan_stages(["read", "map", "reduceByKey", "map"]) is [["read", "map"], ["reduceByKey", "map"]].
6. Distributed processing with Spark
StretchComputationalThe starter computes distinct users per product with a single shuffle keyed by product, so a hot product overloads one partition. Change it to salt the hot products: for a hot product, make the shuffle key (product, salt) with a salt between 0 and S - 1; for the others, keep one salt (for example (product, 0)). Count distinct users per shuffle key, then add the partial counts per product.
The counts must stay exact, and the busiest partition must receive well under its unsalted load. loads must list the number of events each partition receives in the first shuffle.
7. Distributed processing with Spark
CoreHumanOn a cluster with 160 GB of executor memory available for storage, in which situations is cache() likely to make the job faster? Select all that apply.
8. Distributed processing with Spark
CoreHumanFor which of these jobs would you first try one machine with an embedded engine (DuckDB, Polars, DataFusion or similar) rather than a cluster? Select all that apply.
Further reading and resources
- Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing · Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker and Ion Stoica · 2012The NSDI paper that introduced RDDs, lineage and the design behind Apache Spark.
- Spark SQL: Relational Data Processing in Spark · Michael Armbrust et al. · 2015SIGMOD paper describing DataFrames and the Catalyst optimiser.
- Scalability! But at what COST? · Frank McSherry, Michael Isard and Derek G. Murray · 2015A short, sharp comparison of distributed graph systems with a single-threaded laptop program.
- Apache Spark — RDD programming guide
- Apache Spark — SQL performance tuningAdaptive query execution, partition coalescing, join strategy hints and skew-join handling.
- Apache Spark — tuning guideMemory, serialisation and the level of parallelism.
- Dask documentation
- Ray Data documentation
- DuckDB documentation
- Polars user guide