Graceful Shutdown of an asyncio Service

Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. Whatever your process was doing at second 30 is gone. A pool that ignores SIGTERM gets killed mid-write, and you find out about it from half-written files, a queue with items checked out to a worker that no longer exists, and leaked POSIX semaphores that survive the container.

Graceful shutdown is not a nicety. It is the code that decides whether a deploy is invisible or an incident.

The signal handler can only set a flag

loop.add_signal_handler(signal.SIGTERM, service.request_stop)

add_signal_handler is POSIX-only (on Windows you need signal.signal plus a self-pipe), and its callback is a Callable[[], object] — a plain function, not a coroutine function. You cannot await in it, and passing an async def gives you a coroutine object that is created and immediately discarded, doing nothing at all.

So the handler’s entire job is to flip a flag: set an asyncio.Event, and let the code that can actually await react to it. That constraint is why request_stop() is synchronous, and why it must be idempotent — a user who presses Ctrl-C twice, or an orchestrator that sends SIGTERM and then SIGINT, must not produce two shutdowns racing each other.

The ordering rule people get wrong

Cancel producers before consumers, and drain the queue between them.

Reversed, you cancel the consumers while the producers are still pushing, so the queue fills with work that will never be handled — and if the items came from somewhere with at-least-once delivery, they are now checked out to a dead worker and will be redelivered later as duplicates.

The full sequence:

Stop accepting. Close the listener, stop pulling from the source, set the flag. New work is refused; existing work is untouched.
Drain the queue. Let the consumers finish what is already queued, with a bound.
Cancel the consumers that are still running.
Escalate. Cancel again — remember a handler can catch CancelledError once — and after a final window, abandon and return.

Each step gets its own bounded window, so total shutdown time is bounded and you can size it against the orchestrator’s grace period.

💡asyncio.run(main()) does some cleanup on exit. Is that enough? click to reveal

It is necessary and nowhere near sufficient.

asyncio.run finalises async generators — so a suspended async for gets its finally — and shuts down the default thread-pool executor, with a documented five-minute timeout. That last number is worth knowing: if a to_thread call is still blocked, asyncio.run will sit there for up to five minutes, long past any grace period, and then the orchestrator kills you anyway.

What it does not do is cancel your tasks and wait for them. Tasks still pending when the main coroutine returns are simply destroyed, which produces “Task was destroyed but it is pending!” on stderr and leaves whatever they were doing half-done.

So asyncio.run cleans up the runtime; your shutdown path has to clean up your work. The division is deliberate — only you know which of your tasks are worth draining and for how long.

Returning normally matters

A serve() that ends by raising CancelledError looks like a crash to everything above it: asyncio.run propagates it, your exit code is not zero, your orchestrator records a failed termination, and your dashboards show an error where a clean deploy happened.

Shutdown is a successful outcome of a service’s life. The function should return. That means the drain has to catch and absorb the cancellations it caused, and distinguish them from a cancellation that came from outside — which is exactly what asyncio.wait (which never raises for you) plus explicit task.cancel() calls gives you.

💡A handler catches CancelledError and keeps going. How long does shutdown take? click to reveal

As long as your escalation ladder allows, and no longer — provided you built one.

asyncio’s cancellation is edge-triggered, so one cancel() delivers exactly one CancelledError. A handler that catches it and returns to its loop is, from that moment, an ordinary running task again. If your drain does cancel()-then-await once, you wait forever.

The ladder that bounds it: cancel, wait a window, cancel again, wait a window, and then stop waiting. The second cancel catches the handler that swallowed the first; anything that survives two is either malicious or stuck in non-awaiting code, and the only remaining move is to abandon it and let process exit deal with it.

Note what you cannot do: force it. There is no “really cancel” primitive, deliberately — the same reason there is no way to kill a thread. Your shutdown guarantee is therefore “bounded time, best-effort completion”, and the honest thing is to log loudly when you abandon something rather than pretend the drain succeeded.

Typing the state machine

type State = Literal["idle", "running", "draining", "stopped"]

def describe(state: State) -> str:
    match state:
        case "idle": return "not started"
        case "running": return "accepting work"
        case "draining": return "finishing in-flight work"
        case "stopped": return "done"
        case _: assert_never(state)

assert_never makes the match exhaustive at type-check time: add a fifth state and every match that does not handle it becomes an error, pointing at the exact line. Without it, an unhandled state falls through and returns None, and the bug appears somewhere else entirely.

For a shutdown state machine this is not decoration. The states encode which operations are legal, and a missing case is a legal-operation check that silently stopped running.