itertools foundations: chain, islice, and the laziness contract

The whole of itertools rests on one contract: nothing is computed until something asks for it, and nothing is retained after it has been handed over. Every tool in the module is a small state machine that pulls one item from its source when you pull one item from it.

That contract is what makes a pipeline over a 10 GB log file possible in 30 MB of RAM. Break it anywhere — one list(), one sorted(), one len() — and the whole pipeline materialises at that point.

chain, and chain.from_iterable

from itertools import chain

for line in chain(header_lines, body_lines):   # known number of sources
    ...
for line in chain.from_iterable(files):        # sources arrive lazily too
    ...

The difference matters more than it looks. chain(*sources) unpacks sources into positional arguments, which consumes the outer iterable completely before the first item comes out. chain.from_iterable(sources) pulls the next source only when the current one is exhausted. If sources is a generator of open file handles, the first form opens all of them at once.

The alternative — list(a) + list(b) — reads both sides into memory purely in order to walk them once, which is the thing you were trying not to do.

islice

islice(iterable, stop) or islice(iterable, start, stop, step). It is slicing for things that cannot be sliced. Two properties are load-bearing:

• It cannot take negative indices, because that would require knowing the length, which would require consuming the whole source.
• It consumes from the source. islice(it, 5) advances it by five, and the elements it skipped over are gone. When fully consumed with a start and stop, it advances the source by max(start, stop) regardless of step — it has to walk over what it discards.

The composition islice(chain.from_iterable(chunks), n) is a first-n-across- everything that starts no chunk it does not need. That is the shape you want for “give me a preview of this stream”.

💡Why does islice(chain.from_iterable(chunks), 4) over chunks of click to reveal

size 2 start exactly two chunks, not three? Because both layers are demand-driven and neither reads ahead.

chain.from_iterable pulls chunk 1, yields its two items, finds it exhausted, pulls chunk 2, yields its two items. islice has now delivered four items. On the fifth next() call islice checks its counter before pulling from the source, sees it has reached stop, and stops — so chain.from_iterable is never asked for a third item and never pulls chunk 3.

The check-before-pull ordering is the detail. If islice pulled first and then checked, every pipeline would over-consume its source by exactly one element, which for a socket or a cursor is a real bug. The same reasoning explains why islice(it, 0) consumes nothing at all.

The typing surprise

from itertools import chain
reveal_type(chain([1], ["a"]))    # chain[object]

Not chain[int | str]. The element type joins to the nearest common supertype rather than forming a union. For int and str that is object, and everything downstream becomes untyped in practice: x + 1 on the result is an error, x.upper() is an error, and you are left casting.

This is not a bug in typeshed — it falls out of how a single TypeVar is solved across multiple arguments — but it does mean chain over heterogeneous sources is a place where you should stop and declare what you actually want:

merged: Iterator[int | str] = chain(numbers, words)

With the annotation, the checker verifies both sources fit; without it, you get object and no error at all until much later.

💡A colleague replaces list(a) + list(b) with chain(a, b) and click to reveal

the tests start failing with “object of type ‘chain’ has no len()”. What general lesson is hiding in that failure? That laziness is part of a function’s contract, not an optimisation you can apply locally. list supports len, indexing, and repeated iteration; chain supports none of those, and swapping one for the other changes what callers may do with the result.

The two honest resolutions are opposite in spirit. Either commit to laziness and fix the caller — annotate the return as Iterator[T], and replace len() with a running count and indexing with islice — or keep returning a sequence and only use chain internally, materialising at the boundary.

What you must not do is return something lazy from a function annotated -> list[T]. --strict catches that particular lie; it cannot catch the subtler version where you return Iterable[T] and a caller iterates it twice, getting an empty result the second time.