Observability of the runtime itself: an incident playbook keyed by symptom

Every other item in this course teaches you a tool. This one is an index, keyed by the thing you actually have at 02:00: a symptom.

You do not get to start with “I would like to inspect the asyncio task tree.” You start with “requests are timing out and CPU is at 4%”, and the skill is turning that into the right tool in under a minute. That translation is the whole of this page.

The table

Symptom	First move	Then
Hung, CPU near zero, async service	`PYTHONASYNCIODEBUG=1` in a repro; `python -m asyncio ps <pid>` on the live process (3.14+)	Look for a task awaiting something that will never complete, or a `gather` whose sibling died
Hung, CPU near zero, threads	`faulthandler.dump_traceback_later()`, or `kill -SIGABRT` with `faulthandler.enable()`	Lock ordering; a `queue.get()` with no producer left
Hung, one CPU pinned at 100%	Sampling profiler in `cpu` mode	An accidental quadratic, or a regex backtracking
Slow, all cores idle, no obvious blocker	Sampling profiler in `gil` mode (3.15)	Which thread is holding the GIL — the first stdlib answer to that question
Memory grows and never returns	`tracemalloc` snapshot diff; `gc.get_objects()`; memray	A cache with no bound, a logger accumulating records, a task list nobody drains
Memory spikes on one endpoint	`tracemalloc` peak, not current	Materialising a scan before filtering it
Multiprocessing child vanishes	Check the exit code before anything else	OOM killer (137), a segfault in a native extension (139), or a `SystemExit`
Multiprocessing hangs at startup	`pstree` / `ps -ef --forest`	`fork` after threads have started; a missing `if __name__ == "__main__":`
Works locally, hangs in the container	Compare start methods and CPU counts	`fork` vs `forkserver` vs `spawn`; cgroup CPU limits vs `os.cpu_count()`
Fast in isolation, slow under load	Look at the handlers, not the code	A synchronous file or network log handler on the request path

The three tracks this indexes

Asyncio triage. Debug mode (PYTHONASYNCIODEBUG=1) makes the runtime tell you about coroutines that were never awaited, callbacks that blocked the loop for too long, and tasks that were destroyed while pending. 3.14 added python -m asyncio ps and python -m asyncio pstree, which print the task tree of a live process by pid — the single biggest improvement to async incident response in years, because until then the answer to “what is it waiting on” required having thought about it in advance.

Multiprocessing triage. Almost every “multiprocessing is broken” ticket reduces to one of three facts: pickle serialises functions by qualified name so the child re-imports and looks them up; fork copies the memory of a process that may already hold locked mutexes from other threads; and a child that dies does not necessarily tell the parent why. Read the exit code, then the start method, then the picklability of what you sent.

Profiling. PEP 799’s python -m profiling.sampling (3.15) attaches to a running process with no code change, no restart and “zero measurable overhead on the target”. Modes: wall (default), cpu, gil, exception. Outputs include flamegraphs, diff-flamegraphs, a --live TUI and record-and-replay.

Three operational constraints that must be sorted out before the incident, because you will not fix them during one:

Profiler and target must run the same Python minor version — the exact version for pre-releases.
You cannot mix free-threaded and standard builds.
Linux needs root, CAP_SYS_PTRACE, or a relaxed ptrace_scope; macOS needs root or the debugger entitlement; Windows needs SeDebugPrivilege.

For 3.12–3.14 targets the equivalent is py-spy, which attaches the same way and is the reason to keep it in your production image rather than discovering you need it and having no way to install it.

💡CPU sits at 3%, latency is 40× normal, and the sampling profiler in wall mode shows almost all time in select/epoll. What does that tell you, and what does it not? click to reveal

It tells you the process is waiting, not computing. That eliminates half the tools immediately: a CPU profiler will show you nothing interesting, because there is no CPU being spent.

What it does not tell you is what it is waiting for, and that is the actual question. Wall-clock sampling attributes time to the frame that is blocked, which is almost always deep inside the event loop or a socket read — true and useless.

The next move depends on the concurrency model. For asyncio, python -m asyncio ps <pid> gives the task tree with what each task is awaiting; the answer is usually one saturated connection pool, or a downstream that stopped responding and has no timeout. For threads, faulthandler.dump_traceback_later(5) gives you every thread’s stack, and you look for the one holding whatever the others want. For gil mode: it will show almost nothing held, which confirms the diagnosis rather than advancing it.

The general lesson: wall-clock time answers “where is the latency”; CPU time answers “where is the work”. Reaching for the second when the symptom is the first is the most common wasted hour in performance work.

Two things that are invisible to a CPU profiler

Worth stating separately because they cause a lot of wasted profiling:

A cache that never releases. The work is cheap; the memory is not. cProfile shows a fast function. tracemalloc snapshots taken minutes apart show the growth, and a WeakSet canary in a test proves it.
Materialising a scan before filtering it. A list comprehension and a generator expression do the same amount of work. Only the peak memory differs, by three orders of magnitude. No CPU profiler will ever mention it; tracemalloc‘s peak figure names it instantly.

If your only performance tool is a CPU profiler, these two fail as “the box ran out of memory” with no attribution at all.

The rule that makes all of this cheaper

Instrument for the questions you will have, not the answers you have now.

That means: a version-matched profiler already installed in the image; faulthandler.enable() at startup, which costs nothing and turns a segfault into a Python traceback; structured logs with a request id, so “which request” is a filter rather than an archaeology project; and a logging configuration whose handlers are behind a QueueHandler, so that turning up log verbosity during an incident does not add latency to the thing you are debugging.