Debugging Async in Production

“The service is hung. What is it waiting on?”

Before Python 3.14, the honest answer was: add logging and redeploy. Nothing in the standard library could tell you the shape of a live event loop from outside the process. That has changed, and the change is large enough to alter how you operate an async service.

Tier one: debug mode, available everywhere

Three equivalent switches: PYTHONASYNCIODEBUG=1, python -X dev, or asyncio.run(main(), debug=True).

What it buys you:

Wrong-thread API calls raise. Almost everything in asyncio is documented as not thread-safe. In production a cross-thread call_soon usually works, until the day it corrupts the ready queue under load. In debug mode it raises immediately, at the call site, with the offending thread named.

Slow callbacks are logged. Any callback or task step taking longer than loop.slow_callback_duration (default 0.1 seconds) produces a warning naming the callback and the duration. This is the single fastest way to find a blocking call in an async codebase — a synchronous requests.get, a json.dumps of a huge payload, a bcrypt.hashpw — because every one of them shows up as a callback that took far too long.

Un-awaited coroutines get a traceback. Instead of “coroutine ‘x’ was never awaited” with no context, you get the source line where the coroutine was created.

Tasks destroyed while pending are logged, which catches the fire-and-forget GC bug from item 9.8 directly.

The cost is real — extra bookkeeping on every task — so debug mode is for development, CI and incident reproduction, not for steady-state production. Set loop.slow_callback_duration lower than the default when hunting: 0.02 finds things 0.1 hides.

💡Debug mode reports a callback that took 4 seconds. The callback is your own async def handle_request. What have you actually learned? click to reveal

That somewhere inside handle_request, between two awaits, four seconds of uninterrupted work happened — and therefore the entire event loop, every other connection and every timer, was frozen for four seconds.

That framing is what makes the warning useful. A “slow callback” is not slow code, it is code that did not yield. The four seconds are either CPU work or a blocking syscall, and either way the fix is the same shape: move it off the loop with asyncio.to_thread, or break it into chunks with awaits between them.

What it does not tell you is which line. For that: run it again with -X importtime-style bisection, or attach py-spy dump while it is happening, or — on 3.14 — use the call-graph tooling below. The warning is a smoke alarm, not a map.

Tier two: 3.14’s live introspection

This is the change. Against a running process, with no code modification and no restart:

python -m asyncio ps <PID>
python -m asyncio pstree <PID>

ps gives a flat table of every task with its coroutine stack. pstree renders the await graph as a tree: which task is awaiting which, all the way down to the coroutine and frame that is actually blocked. For a hung service that is the entire question, answered in one command.

It also detects cycles in the await graph and reports them as an error rather than hanging while it walks them — so the classic deadlock (task A awaits B, B awaits A) is named rather than merely observed.

In-process, the same machinery is available as asyncio.capture_call_graph() and asyncio.print_call_graph(), which is what you call from a signal handler or a /debug/tasks endpoint to dump the loop’s state into your own logs.

Python 3.15 builds CPython with frame pointers by default (PEP 831), which makes native stack unwinding reliable — so the external profilers and debuggers that had to guess at frames get accurate ones.

💡Your service is hung in production on Python 3.12, where python -m asyncio pstree does not exist. What is the equivalent, and what does it cost? click to reveal

Two useful approximations, neither free.

py-spy dump --pid <PID> gives you the native stack of every OS thread without touching the process. On an async service that shows you the event loop thread parked in epoll_wait and not much else — because every suspended task is a heap object, not a stack frame. It answers “is the loop blocked in Python code?” (which the 4-second-callback question above needs) but not “which task is waiting on what”.

For the task graph you have to have planned ahead: a signal handler that dumps asyncio.all_tasks() with task.get_stack() and task.get_name() into your logs. Twenty lines, registered on SIGUSR1, costing nothing at rest. If you run async services on 3.12 or 3.13, write it before you need it — because the one time you want it is the one time you cannot deploy.

Name your tasks, too. asyncio.create_task(coro, name="refresh:tenant-42") is free, and the difference between a dump full of Task-17 and one full of meaningful names is the difference between a diagnosis and a list.

What to do on Monday

Turn on debug mode in your test suite. asyncio.run(main(), debug=True) in fixtures costs nothing and turns “flaky in CI” into “raises at the call site”.
Name every task you create.
On 3.14+, know that python -m asyncio pstree <PID> exists before you need it. On 3.12/3.13, write the SIGUSR1 dump handler now.
Lower slow_callback_duration when you are hunting a latency spike, not when you are not.