Skip to content
LumenaryAcademy
Foundations 55 min

Iteration, generators and comprehensions

The protocol behind every for loop, functions that pause, and the toolbox for working with streams of data.

Code tested · 7AI-draftedAwaiting review by a computer scientist
MotivationH

The loop that works on everything

A for loop in Python works over a list, a string, a dictionary, a file, a range of a billion numbers, the rows of a database query and a stream of messages from a network socket. Some of those already sit in memory; some do not exist yet when the loop starts; one of them might never end. Yet the loop looks the same every time.

That uniformity comes from the iteration protocol, two special methods that are among the most important in the language. On top of it Python builds generators, functions that produce values one at a time and pause in between, and comprehensions, the compact syntax for building collections that you will see in almost every Python file.

Together they let you write programs that process data as a stream: read one record, transform it, filter it, aggregate it, and only then read the next. A program written that way uses the same small amount of memory for ten lines or ten billion, and it starts producing results immediately. This is one of the most practical skills in professional Python, whether you are parsing logs, reading a large CSV export or paging through an API.

DefinitionsH

Iterables and iterators

Definition— Iterable

An object you can loop over: it has an __iter__ method that returns an iterator (or, in the old protocol from lesson 2, a __getitem__ that accepts 0, 1, 2, …). Lists, tuples, strings, dicts, sets, files and ranges are iterable.

Definition— Iterator

An object that produces the items, one per call to its __next__ method, and raises StopIteration when there are no more. An iterator is also iterable: its __iter__ returns itself.

The built-in iter(x) calls x.__iter__() and next(it) calls it.__next__(). A for loop is just these two calls and an exception handler:

python
"""What a for loop really does."""
colours = ["ochre", "indigo", "jade"]

it = iter(colours)              # ask the iterable for a fresh iterator
print(next(it), next(it), next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration: this iterator is used up")

# for colour in colours: print(colour.upper()), spelled out:
it = iter(colours)
while True:
    try:
        colour = next(it)
    except StopIteration:
        break
    print(colour.upper())


class CountdownIterator:
    """A hand-written iterator: it remembers where it is."""
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self              # iterators are their own iterators

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1


countdown = CountdownIterator(3)
print(list(countdown))
print(list(countdown))           # the same iterator again: nothing left
print(list(colours), list(colours))  # a list hands out a new iterator each time
text
ochre indigo jade
StopIteration: this iterator is used up
OCHRE
INDIGO
JADE
[3, 2, 1]
[]
['ochre', 'indigo', 'jade'] ['ochre', 'indigo', 'jade']

The last three lines show the distinction that matters in practice. A list is an iterable that gives out a new iterator every time you loop, so you can loop over it as often as you like. An iterator is a one-way, one-time cursor. Once it is exhausted, looping over it again silently produces nothing. File objects, generators, map, filter, zip and enumerate all return iterators, which is why code that loops over one of them twice is a classic bug.

Why have two concepts? Because the cursor's position must live somewhere. Keeping it in a separate iterator object lets two loops walk the same list at the same time, each at its own pace, and lets objects such as files, which really are one-way streams, say so honestly.

IntuitionH

Generators: functions that pause

Writing __iter__ and __next__ by hand works, but it means storing all the state in attributes by hand. A generator function does it for you. It is any function containing yield. Calling it does not run its body; it returns a generator object, which is an iterator. Each next() runs the body until the next yield, hands out that value, and freezes the function, local variables and all, until the next request.

python
def countdown(n):
    print("  (starting)")
    while n > 0:
        yield n
        n -= 1
    print("  (finished)")


gen = countdown(2)
print(type(gen).__name__)       # nothing has run yet
print(next(gen))
print(next(gen))
print(next(gen, "done"))        # next's second argument is a default for exhaustion


def flatten(items):
    """Yield the leaves of arbitrarily nested lists."""
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)   # delegate to a sub-generator
        else:
            yield item


print(list(flatten([1, [2, [3, 4]], [], 5])))
text
generator
  (starting)
2
1
  (finished)
done
[1, 2, 3, 4, 5]

Notice that "(starting)" was printed only when the first value was requested, and "(finished)" only when the third request found the loop over and the function returned; returning raises StopIteration for you. yield from other (PEP 380, Python 3.3) yields everything other produces, which makes recursive generators like flatten read naturally.

Pipelines

Generators shine when you chain them. Each stage pulls items from the previous one, so data flows through the whole pipeline one item at a time. Here a "file" (an in-memory one, so the example is self-contained) is cleaned, parsed and filtered without ever building a list:

python
import io

LOG = io.StringIO("""\
2026-03-01 Nairobi 23.5
2026-03-01 Oslo -4.0
# sensor 7 offline
2026-03-02 Nairobi 24.1

2026-03-02 Oslo -2.5
2026-03-03 Nairobi n/a
2026-03-03 Oslo 0.5
""")


def meaningful_lines(f):
    for line in f:                       # files are iterators over their lines
        line = line.strip()
        if line and not line.startswith("#"):
            yield line


def readings(lines):
    for line in lines:
        date, city, value = line.split()
        try:
            yield date, city, float(value)
        except ValueError:
            continue                     # skip malformed values


above_zero = (r for r in readings(meaningful_lines(LOG)) if r[2] > 0)
for date, city, celsius in above_zero:
    print(f"{date} {city:<8}{celsius:5.1f}")
text
2026-03-01 Nairobi  23.5
2026-03-02 Nairobi  24.1
2026-03-03 Oslo      0.5

If LOG were a 40 GB file opened with open(), this program would use the same memory and start printing at once. Each stage is a small function you can test on its own, and adding a stage (deduplicate, convert units, stop after 100 results with itertools.islice) is one more line.

The expression in parentheses, (r for r in ... if ...), is a generator expression: the lazy cousin of a list comprehension. When it is the only argument to a function you can drop the extra parentheses: sum(x * x for x in data).

Worked examplesH

Comprehensions

A comprehension builds a list, set or dict from an iterable in one expression. The shape is always expression, then one or more for clauses, then optional if filters, read left to right exactly like the nested loops they replace. Run this and change it:

python
import sys

words = ["sol", "luna", "estrella", "mar", "cielo", "luna"]

lengths = [len(w) for w in words]                 # list comprehension
loud = [w.upper() for w in words if len(w) > 3]   # with a filter
unique = {w for w in words}                       # set comprehension
position = {w: i for i, w in enumerate(words)}    # dict comprehension
print(lengths)
print(loud)
print(sorted(unique))
print(position)                                   # a repeated key keeps its last value

# Two for clauses read like two nested loops, outer first
pairs = [(x, y) for x in range(3) for y in range(3) if x < y]
print(pairs)

# Square brackets build the whole list; parentheses make a lazy generator
squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))
print(sys.getsizeof(squares_list) > 1_000_000, sys.getsizeof(squares_gen) < 1_000)
print(sum(n * n for n in range(1_000_000)))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
text
[3, 4, 8, 3, 5, 4]
['LUNA', 'ESTRELLA', 'CIELO', 'LUNA']
['cielo', 'estrella', 'luna', 'mar', 'sol']
{'sol': 0, 'luna': 5, 'estrella': 2, 'mar': 3, 'cielo': 4}
[(0, 1), (0, 2), (1, 2)]
True True
333332833333500000

Some guidance that experienced reviewers apply:

  • One idea per comprehension. A single for with an optional if is almost always clearer than the equivalent loop. Two for clauses are fine for something like pairs. Beyond that, or with a condition that needs a comment, write a loop or a helper function.
  • Comprehensions build things. Do not write [print(x) for x in items] to get side effects; it builds a useless list of Nones. Use a for statement.
  • Choose the bracket for the job. If the result is consumed once, by sum, max, any, "".join or a for loop, a generator expression avoids building a list. If you need len, indexing or several passes, build the list.
  • Unpacking arrived in 3.15. Python 3.15 (PEP 798) lets a comprehension unpack each item: [*row for row in rows] flattens a list of lists, and {**d for d in dicts} merges dictionaries. On earlier versions, write two for clauses or use itertools.chain.from_iterable.
  • The loop variable stays inside. Unlike a for statement, a comprehension's variable does not leak into the surrounding scope. (Since Python 3.12, list, set and dict comprehensions are also inlined by the compiler, PEP 709, which made them faster.)
ComputationC

itertools and the iteration built-ins

The built-ins you should know cold: enumerate, zip, map, filter, reversed, sorted (with key=), min and max (with key= and default=), sum, any and all. The itertools module adds fast, lazy building blocks for streams. A quick tour:

python
import itertools as it

print(list(it.islice(it.count(10, 5), 4)))       # slices of any iterator, even an infinite one
print(list(it.chain("ab", [1, 2])))               # one stream after another
print(list(it.accumulate([3, 1, 4, 1, 5])))       # running totals (or any binary function)
print(list(it.pairwise("ABCD")))                  # neighbours (3.10+)
print(list(it.batched(range(7), 3)))              # fixed-size chunks (3.12+)
print(list(it.combinations("XYZ", 2)))
print(len(list(it.product(range(3), repeat=2))))  # every pair: 3 * 3
print(list(it.zip_longest("abc", [1], fillvalue="-")))
print(list(it.takewhile(lambda x: x < 4, [1, 3, 5, 2])))

words = ["apple", "avocado", "banana", "blueberry", "cherry", "apricot"]
for letter, group in it.groupby(sorted(words), key=lambda w: w[0]):
    print(letter, list(group))

print(max(words, key=len), min([], default="(empty)"))
print(any(w.startswith("c") for w in words), all(len(w) > 4 for w in words))
text
[10, 15, 20, 25]
['a', 'b', 1, 2]
[3, 4, 8, 9, 14]
[('A', 'B'), ('B', 'C'), ('C', 'D')]
[(0, 1, 2), (3, 4, 5), (6,)]
[('X', 'Y'), ('X', 'Z'), ('Y', 'Z')]
9
[('a', 1), ('b', '-'), ('c', '-')]
[1, 3]
a ['apple', 'apricot', 'avocado']
b ['banana', 'blueberry']
c ['cherry']
blueberry (empty)
True True

Two of these deserve a warning. groupby groups consecutive equal keys, like the Unix uniq command, so you almost always sort by the same key first; without sorted, the two "a" groups in words would come out separately. And batched is new in 3.12: on older versions the itertools documentation gives a short recipe, and lots of projects carry their own copy.

any and all short-circuit: any stops at the first true item and all at the first false one. Combined with a generator expression, that means they do no more work than necessary, which matters when each item is expensive to compute.

ExplanationH

Laziness in real code

When should a function return a list, and when a generator? Professionals tend to follow a few rules of thumb.

Return a list when the result is small, when callers will want len() or indexing, or when they will iterate more than once. A list is also easier to debug: you can print it.

Return a generator (or accept any iterable) when the data is large or unbounded, when it comes from I/O such as a file, a database cursor or a paginated API, or when callers may stop early. Accepting any iterable as input, instead of insisting on a list, makes a function work with all of the above for free; it is why sum, sorted and "".join are so widely useful.

Mind the lifetime of resources. A generator that reads from an open file must not outlive the with block that opened it. A common pattern is to keep the with inside the generator function, so the file closes when the generator finishes or is garbage-collected.

Generators can stop early and clean up. A generator that is abandoned half-way, because the caller broke out of its loop or islice took only what it needed, is closed: Python raises GeneratorExit at the paused yield, so any finally block or with block inside the generator runs. The next section shows this happening.

ComputationC

Generators that clean up and listen

A paused generator is a suspended function call, with its local variables and its position saved. Two methods let the caller reach into that paused call.

gen.close() raises GeneratorExit at the yield where the generator is paused. The generator is not supposed to catch it; it should let it propagate, running any finally clauses on the way out. That is how a generator holding a file or a network connection releases it when its consumer loses interest.

gen.send(value) resumes the generator and makes the paused yield expression evaluate to value. So received = yield result both hands a value out and takes one in. A generator used this way is a coroutine: a function that runs a little, waits for input, and runs again, keeping its state between steps. Before sending anything you must advance it to its first yield with next(gen), which is called priming.

python
def numbered(name):
    print(f"  open {name}")
    try:
        n = 0
        while True:
            n += 1
            yield f"{name}:{n}"
    finally:
        print(f"  close {name}")    # runs however the generator ends


lines = numbered("orders.csv")
print(next(lines), next(lines))
lines.close()                       # GeneratorExit at the paused yield; finally runs
print(next(lines, "nothing more"))


def averager():
    """A coroutine: send it numbers, get back the running mean."""
    total = count = 0
    mean = None
    while True:
        value = yield mean          # hand out the mean, wait for the next value
        total += value
        count += 1
        mean = total / count


avg = averager()
next(avg)                           # prime: run to the first yield
print(avg.send(10), avg.send(20), avg.send(60))
text
  open orders.csv
orders.csv:1 orders.csv:2
  close orders.csv
nothing more
10.0 15.0 30.0

The averager shape, a loop around value = yield result, was how people wrote asynchronous code with generators in the late 2000s. PEP 342 (2005) added send and close; PEP 380 (2012) added yield from, which also forwards send calls to a sub-generator; and in 2015 PEP 492 gave coroutines their own async def and await syntax. You rarely call send in application code today, but when lesson 9 explains how await suspends a function, you will already know the mechanism underneath.

History & global context

Where these ideas came from

Almost nothing in this lesson was in the first Python. Iteration grew one PEP at a time, and each step borrowed from another language.

  • List comprehensions arrived in Python 2.0 (2000, PEP 202), with syntax borrowed from Haskell, whose own notation imitates the set-builder notation of mathematics: {x2∣x∈S, x>3}\{x^2 \mid x \in S,\ x > 3\}.
  • The iterator protocol (__iter__, __next__) and generators came in Python 2.2 (2001, PEPs 234 and 255). The generator PEP credits the Icon language, where expressions that produce a sequence of results were central.
  • itertools appeared in Python 2.3; its documentation says its building blocks were inspired by APL, Haskell and SML.
  • Generator expressions followed in 2.4 (PEP 289), and set and dict comprehensions in 3.0 and 2.7.
  • Coroutines went from send (PEP 342) and yield from (PEP 380) to async/await (PEP 492, Python 3.5).

Each step generalised the one before. That is a recurring pattern in Python's history and a good reason to read the PEPs: the "Rationale" and "Rejected ideas" sections explain trade-offs that the documentation, which only describes the final design, leaves out.

Common misconceptionsH

Common misconceptions

Misconception

"If a loop runs zero times, the data must have been empty." Or the iterator was already exhausted by an earlier loop, a list() call, an in test or a debugging print(list(it)). Exhausted iterators fail silently. If you need two passes, build a list first, or create the iterator again.

Misconception

"I can remove items from a list while looping over it." The list's iterator walks by position, so removing an item shifts the rest left and the next one is skipped. Build a new list with a comprehension, [x for x in xs if keep(x)], or loop over a copy.

Misconception

"(x for x in data) is a tuple comprehension." There is no tuple comprehension: parentheses give a generator expression. For a tuple, write tuple(x for x in data).

Misconception

"Generators make code faster." They save memory and let work start sooner. For a small collection you will loop over several times, a list is often slightly faster. Choose by the shape of the data, and measure if speed matters (lesson 10).

Exercises

Exercises

Several of these probe exactly when an iterator is consumed. Predict first, then run to check. The last exercise asks you to write a lazy generator.

1. Iteration, generators and comprehensions

CoreComputational

What does the second print show?

evens = (n for n in range(10) if n % 2 == 0)
print(4 in evens)
print(list(evens))

2. Iteration, generators and comprehensions

StretchComputational

What does this program print?

it = iter([1, 2, 3, 4, 5])
print(list(zip(it, it)))

3. Iteration, generators and comprehensions

CoreComputational

How many lines starting with computing does this program print?

def check(n):
    print("computing", n)
    return n > 2


print(any(check(n) for n in [1, 3, 5, 7]))

4. Iteration, generators and comprehensions

CoreComputational

What does this program print?

from itertools import groupby
print([key for key, _ in groupby("aabbaa")])

5. Iteration, generators and comprehensions

Warm-upComputational

What does this program print?

print({len(w): w for w in ["ox", "cat", "be", "dog"]})

6. Iteration, generators and comprehensions

CoreComputational

Which of these objects are iterators (so iter(x) is x, and looping over them a second time yields nothing)? Choose all that apply.

Choose all that apply

7. Iteration, generators and comprehensions

StretchComputational

Write a generator function chunked(iterable, size) that yields lists of size consecutive items from any iterable; the last list may be shorter. It must be lazy: it must work on an infinite iterator such as itertools.count(), pulling only the items it needs.

list(chunked(range(7), 3))   # [[0, 1, 2], [3, 4, 5], [6]]

(Python 3.12 has itertools.batched, which yields tuples. Write your own without it, so it works on older versions too.)

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources