Generators as a memory strategy

A batch job dies at 3am. The traceback is MemoryError, or there is no traceback at all because the kernel’s OOM killer got there first. Someone reads the file that morning and finds this:

rows = [parse(line) for line in open(path)]
for row in rows:
    handle(row)

The fix is one character. The understanding is not.

The measurement

On CPython 3.14 (macOS arm64), tracemalloc over a million squares:

import tracemalloc

tracemalloc.start()
eager = [n * n for n in range(1_000_000)]
print(tracemalloc.get_traced_memory())   # peak ≈ 40.4 MB

tracemalloc.reset_peak()
lazy = (n * n for n in range(1_000_000))
print(tracemalloc.get_traced_memory())   # peak ≈ 0.0 MB

40.4 MB versus 0.0 MB. Same arithmetic, same answers in the same order, four orders of magnitude apart on peak allocation.

The second number is not “small”. It is nothing: the generator expression allocated a generator object and stopped. No squares have been computed yet. Not one.

What actually changed

Nothing about the arithmetic. What changed is who drives.

A list comprehension is a loop that runs to completion and hands you the answers. A generator is an object with a suspended stack frame, and it computes exactly one value per next() call, then suspends again with its locals intact.

def squares(n: int) -> Iterator[int]:
    for i in range(n):
        yield i * i        # <- frame suspends here, locals preserved

Calling squares(1_000_000) executes none of that body. Not the for, not range, not the first multiply. It builds a generator object and returns. This is the single most important thing to internalise, because everything else follows from it:

• The for loop has not started, so nothing has been read.
• Any argument validation you wrote at the top of the function has not run.
• Any exception the body would raise has not been raised.
• Any side effect — a log line, an open file, a metric increment — has not happened.

Laziness moves when, not just where

This is the part that bites. Consider a function that looks like it validates its input:

def parse_all(lines: Iterable[str], *, strict: bool) -> Iterator[Record]:
    if strict and not lines:
        raise ValueError("no lines")     # <- never runs at call time
    for line in lines:
        yield parse(line)                # <- raises here, later, elsewhere

parse_all([], strict=True) raises nothing. The ValueError is deferred until someone iterates — which may be inside a JSON serialiser, inside a template renderer, inside a logging handler, three call frames away and half a second later. The traceback names the consumer, not the producer.

Two consequences worth building habits around:

Validate eagerly with a wrapper. If an argument is wrong, say so at the call site. Split the function in two: a plain def that checks and returns, and the generator that yields.

def parse_all(lines: Iterable[str], *, chunk: int) -> Iterator[list[Record]]:
    if chunk <= 0:
        raise ValueError(f"chunk must be positive, got {chunk}")
    return _parse_all(lines, chunk)     # plain def: this raises NOW


def _parse_all(lines: Iterable[str], chunk: int) -> Iterator[list[Record]]:
    ...                                  # the generator

Beware laziness across a resource boundary. This is the classic:

def read_rows(path: str) -> Iterator[str]:
    with open(path) as handle:
        yield from handle

The with block does not close the file when read_rows returns — it returns immediately, having done nothing. The file opens on the first next() and closes when the generator is exhausted or garbage-collected. Break out of the loop early and the file stays open until the collector notices. On CPython that is usually “soon”; on PyPy or under a reference cycle it is “eventually”; in a service holding 4,000 file descriptors it is an outage.

from contextlib import closing

with closing(read_rows(path)) as rows:
    for row in rows:
        if row.startswith("#"):
            break            # generator.close() runs at the end of the with

Single consumption is a contract, not a quirk

A list can be read as many times as you like. A generator cannot be read twice — and the second read does not fail loudly. It yields nothing.

rows = (parse(line) for line in source)
count = sum(1 for _ in rows)     # 1_000_000
total = sum(r.amount for r in rows)   # 0.  No error. No warning.

This is the quietest bug in the iteration protocol, and it exists because an iterator is its own iterable: iter(gen) is gen. It satisfies Iterable[T] perfectly while being single-shot, so no annotation you write on the parameter catches it.

The discipline: a function that takes Iterable[T] must traverse it at most once. If you need two passes, either say so in the signature by asking for Sequence[T], or materialise deliberately and say why in a comment. Do not reach for itertools.tee as a reflex — tee buffers whatever the lagging iterator has not consumed, so if one copy is drained before the other starts, the buffer holds the whole stream anyway. The standard library’s own documentation says that if one iterator will use most of the data before another starts, list() is faster.

Where laziness costs you

Laziness is not free and it is not always right.

For a thousand rows, materialise. The clarity is worth more than the bytes. Laziness earns its keep when the data is large relative to memory, when the source is unbounded, when you want the first result before the last one is computed, or when a downstream stage may stop early.

Batching: the shape you actually ship

Pure element-at-a-time streaming is often not what you want either, because the thing downstream — a database, an HTTP API, a message broker — has a per-call cost you would rather amortise. The production shape is a lazy stream of bounded batches:

from collections.abc import Iterable, Iterator


def in_batches[T](items: Iterable[T], size: int) -> Iterator[list[T]]:
    batch: list[T] = []
    for item in items:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:            # the remainder — the line everyone forgets
        yield batch

Peak retention is size items, whatever the input length. That is the property to state in review: not “it uses generators”, but “peak retention is bounded by size“.

Since 3.12 the standard library has itertools.batched, which does the same thing lazily and yields tuples rather than lists. Prefer it unless you specifically need mutable batches.

The annotation is the contract

Finally, annotate honestly. A function containing yield does not return list[Record], and writing that annotation is a lie mypy will catch:

def parse_records(lines: Iterable[str]) -> list[Record]:   # wrong
    for line in lines:
        yield parse(line)

error: The return type of a generator function should be "Generator" or one of its supertypes  [misc]

Iterator[Record] is the right annotation for a generator whose send() and return values nobody uses — which is almost all of them. Generator[Record, None, None] is the same thing spelled out; reach for it only when the extra parameters carry information. Import from collections.abc, not typing: the typing aliases have been deprecated since 3.9, though removal is not currently planned. Note that mypy --strict will not flag typing.List — modernisation is ruff’s job (rule UP035), not the type checker’s.