Skip to content

← Generators, Iterators and Streaming Data step 7 of 9

Medium Primitives

Copy a One-Shot Iterator into Separate Lists

Two parts of your program need the same records, and the records come from a generator, so you can read them only once. The article Split an Iterator with tee and Combine Inputs with product gave the simple fix for the case where each part reads everything. Read the stream into a list once, and loop over the list. Here you write that helper. It returns one list per part, and it refuses a stream longer than a cap, so a huge or endless stream cannot fill your memory.

tee_safely(iter([1, 2]), 2, cap=2)  # ([1, 2], [1, 2])
tee_safely(iter([1, 2]), 2, cap=1)  # ValueError: source exceeds the 1-item cap

What to write

Fill in tee_safely(source, n, *, cap). The tests always pass a cap of zero or more.

  • If n is negative, raise ValueError(f"n must be non-negative, got {n}"). Check this first, before you call iter() on source or read from it.
  • Otherwise loop over source once, and read at most cap + 1 items.
  • If you read cap + 1 items, the source is longer than the cap. Raise ValueError(f"source exceeds the {cap}-item cap") right away, and read nothing more.
  • If the source fits, return a tuple of n lists. Each list holds the items in their original order.
  • Make each list a separate object, so appending to one leaves the others alone. All the lists hold the same item objects, so if an item is itself a list, a change to it shows in every copy.
  • Return a tuple of lists, not a list of lists.

Edge cases

  • n == 0 still reads the source and checks the cap. If the source fits, the result is ().
  • cap == 0 accepts an empty source. Any other source fails after you read one item.
  • An empty source gives n empty lists.

Fill in solve

Keep CountingSource, Report and _independent as they are. CountingSource wraps an iterable. It counts how many times something calls iter() on it (iter_calls) and how many items it hands out (pulled).

In solve, use items as the source, or (x for x in items) when lazy is True. Wrap it in CountingSource, call tee_safely once, and catch ValueError. Return a report with these keys:

  • copies: the tuple from tee_safely, or () if it raised.
  • independent: _independent(copies) on success, and True on failure. _independent appends to the first list, checks that the second did not change, then removes the item again. It returns True for zero or one copy.
  • iter_calls: the CountingSource count, read after the call, whether it returned or raised.
  • pulled: the CountingSource count, read the same way.
  • error: "" on success, or the message of the ValueError.

A negative n gives 0 for both counts. Every other call gives iter_calls == 1, even for an empty source or n == 0. On success, pulled is the length of items. When the source is too long, it is cap + 1.

For items=[1, 2], n=2 and cap=1, solve returns copies=(), independent=True, iter_calls=1, pulled=2 and error="source exceeds the 1-item cap".