Build a Paginator You Can Close Early

Build Paginator, an async iterator over an API that returns results one page at a time. It fetches the next page only when the caller needs another item, and the caller can close it early.

# fetch(0) returns [1, 2], fetch(1) returns [3], fetch(2) returns []
paginator = Paginator(fetch)
async with aclosing(paginator):
    async for item in paginator:
        print(item)  # 1, then 2, then 3

fetch(n) is a coroutine function that returns page n as a list. An empty page means there are no more pages.

What to write

The starter already has the constructor. It stores fetch, sets _page to 0, makes an empty deque called _buffer, and sets _exhausted and closed to False. Fill in three methods:

• __aiter__() is a plain def that returns self. It changes nothing.
• async def __anext__() returns the next item:
  • If _buffer holds an item, remove the first one and return it.
  • Otherwise, if closed or _exhausted is True, raise StopAsyncIteration without fetching.
  • Otherwise, await self._fetch(self._page). When the fetch succeeds, add one to _page. Put the items of a nonempty page in the buffer and return the first one. For an empty page, set _exhausted to True and raise StopAsyncIteration.
• async def aclose() sets closed to True and empties the buffer. A second call changes nothing. After it, every __anext__() raises StopAsyncIteration without fetching.

Also follow these rules:

• Fetch only when the buffer is empty and the caller asks for an item. Never fetch ahead.
• An empty page ends the stream for good, even if later pages have items. Don't skip over it.
• If fetch raises, let the error propagate and leave _page as it was, so a retry asks for the same page.

What the driver reports

Keep _drive and solve as they are. The driver's fetch returns a copy of pages[n], or [] past the last page, and counts its calls. The driver loops over your paginator inside async with aclosing(paginator). If the test passes stop_after, the loop breaks once it has that many items. After the block, the driver calls await anext(paginator, sentinel) once more.

• items: the items the loop collected, in order.
• fetch_calls: how many times fetch ran, counting the call that returned the empty page.
• closed: paginator.closed after the block. aclosing called your aclose(), so this is True.
• after_close: the result of that last anext. A closed paginator stops, so this is sentinel, which is -1 unless the test sets it.

With pages=[[1, 2], [3]], reading everything gives [1, 2, 3] after 3 fetches, two for the pages and one for the empty page. With stop_after=2 it takes 1 fetch, and with stop_after=3 it takes 2, because the loop stops before it asks for a fourth item.

With pages=[] there is still 1 fetch. With pages=[[1], [], [2]] the stream ends at the empty middle page, so the paginator never fetches [2].