Tasks, Task Lifetime, and the Fire-and-Forget GC Bug

Two incident reports that turn out to be the same bug:

“Our background refresh randomly stops running under memory pressure.”

“We found this exception in the logs six hours later, with no context.”

Both come from asyncio.create_task(...) on a line by itself.

The documented hazard

The stdlib says it in bold:

Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage-collected at any time, even before it’s done.

So asyncio.create_task(refresh()) discards the only strong reference the task had. Whether it survives depends on whether a garbage collection happens to run during a window in which nothing else refers to it — which depends on allocation pressure, which depends on load. That is why it works in tests and fails in production, and why it fails more the busier you are.

The prescribed fix is three lines:

_background: set[asyncio.Task[None]] = set()

def spawn(coro: Coroutine[Any, Any, None]) -> asyncio.Task[None]:
    task = asyncio.create_task(coro)
    _background.add(task)
    task.add_done_callback(_background.discard)
    return task

The add_done_callback is not optional in the other direction either: without it the set grows forever, and a long-lived service leaks one Task — plus its frames, plus everything they close over — per spawn.

This is still the documented behaviour in 3.15. Python 3.14 rewrote task bookkeeping for performance (a per-thread linked list, current task on the thread state) but did not change the ownership contract; the issue asking for strong references, gh-91887, remains open. Do not assume a version bump fixed it.

💡Why does the event loop hold only weak references? It could hold strong ones and this whole class of bug would vanish. click to reveal

Because “the loop owns every task” and “an abandoned task is a leak” are the same statement viewed from opposite ends.

If the loop held strong references, a task that nothing will ever await, whose result nobody wants, and whose coroutine is blocked forever on a queue that will never be fed, would be kept alive for the process’s lifetime — along with its frames and everything they close over. There would be no way for the runtime to distinguish “abandoned” from “deliberately long-lived”, so it would have to keep everything.

Weak references push the decision to you: something has to own a task, and the runtime declines to guess what. That is the same design as threading, where a Thread object you drop on the floor keeps running because the threading module holds it — the difference being that asyncio made the opposite choice, and only documented it.

Whether that was the right call is genuinely debated (gh-91887 is open for exactly this reason). What is not debated is what the current contract is.

The second half: the exception nobody sees

A Task stores its exception and waits for someone to ask. If nobody ever awaits it, the exception surfaces at garbage-collection time, via the loop’s exception handler, as:

Task exception was never retrieved
future: <Task finished coro=<refresh() done> exception=ValueError(...)>

That is the “six hours later, no context” line. The timestamp is when the GC ran, not when the failure happened. The traceback has no request id, no user, no correlation with the log lines around it — because the code that spawned it returned long ago.

A BackgroundTasks object with a drain() is what converts that into a normal exception at a place you chose. And if a task genuinely is fire-and-forget — nobody will ever await it — then the done_callback is where you log the failure yourself, deliberately, with context, rather than leaving it to the collector.

💡asyncio.shield(coro) carries the same hazard. Why, and what does it mean for the "shield the cleanup" pattern? click to reveal

shield wraps its argument in a Task internally, and the loop holds only a weak reference to that inner task, exactly as with create_task. If the outer awaiter is cancelled and nothing else holds the shielded task, the thing you shielded can be collected — so the operation you specifically protected from cancellation is the one that disappears.

For “shield the cleanup”, it means the pattern needs the same discipline: assign the shielded task to a name that outlives the scope, or it is not actually protected.

The broader point, which item 9.10 makes at length, is that shield is usually the wrong answer anyway. The docs’ own example is captioned “completely ignore cancellation (not recommended)”. If your cleanup must survive cancellation, the tool is a bounded finally with its own timeout — not a construct that opts out of the cancellation protocol and quietly reintroduces the weak-reference problem.