Skip to content

← Stdlib Mastery step 15 of 55

Easy Primitives

batched: page a stream, and validate before you yield

Turn a stream of records into pages, and reject a bad page size at call time, not at first iteration.

def bulk_pages(records: Iterable[str], size: int) -> Iterator[list[str]]: ...
def solve(records: list[str], size: int) -> dict[str, object]: ...
  • bulk_pages yields list[str] pages of size records, with a shorter final page when the stream does not divide evenly.
  • size < 1 must raise ValueError from the call itself, before anything is iterated.
  • Never hold more than one page at a time. Take Iterable[str], not list[str].

solve reports which of the two moments raised:

{"raised_on_call": bool, "raised_on_iter": bool, "pages": list[list[str]]}

It calls bulk_pages(iter(records), size) inside one try, then list(...) the result inside a second one.

This is the assertion. A generator function defers its entire body to the first next(), including the validation at the top. bulk_pages(records, 0) returns a generator object quite happily and the ValueError fires later — often in a different function, several frames from the bad argument, with a traceback that points at the consumer. In a pipeline assembled in one place and consumed in another that is a genuinely hard bug to read. The fix is the standard shape: a plain function that validates and returns an inner generator. The return type does not change, so no caller is affected.

Why not range(0, len(records), size). That version needs len() and slicing, so it works on a list and breaks on a generator, a DB cursor, a file handle or a paginated API — the entire set of cases where you needed to batch in the first place, because they are the ones too large to hold in memory. itertools.batched (3.12) works on any iterable.

A typing note you are not being tested on but should know. batched yields tuple[T, ...] — variadic, not fixed-length — so for a, b in batched(xs, 2) type-checks and explodes on a short final batch. strict=True is the runtime fix, and it is 3.13+; the checker cannot help on either version.

Loading visualization…