Your service stops answering requests, and its CPU use is near zero. Restarting it would bring it back, but a restart also throws away the evidence. The cheapest evidence is the stack of every thread, and one call at startup lets you ask for it at any time:
import faulthandler
import signal
faulthandler.register(signal.SIGUSR1)
On Linux or macOS, kill -USR1 PID now makes the process print every
thread's stack to standard error and keep running. Replace PID with the
process ID. Here is the first part of the output for a thread stuck on a lock,
on Python 3.15, with the paths shortened:
Thread 0x00000001701d7000 [worker] (most recent call first):
File ".../service.py", line 11 in handle_request
File ".../threading.py", line 1160 in run
Line 11 is the with lock: that the thread is waiting at. Make sure standard
error goes to a log you keep, or you lose the dump.
Start from the symptom
This table matches a symptom to the first evidence to collect, and to the article earlier in the roadmap that shows the tool.
| Symptom | First evidence to collect | Read more |
|---|---|---|
| Async requests stall and CPU is low | Every pending task and what it awaits | Debug a Slow or Stuck Async Service |
| Threads seem stuck | Two or three stack dumps, a few seconds apart | This article |
| One core stays busy | A sampling profile taken while it is slow | The sampling profiler article |
| Memory grows with every request | Two tracemalloc snapshots, compared |
The tracemalloc article |
| One operation makes memory jump | The peak from tracemalloc.get_traced_memory() around that operation |
The tracemalloc article |
| A worker process disappears | Its exit code, its standard error and the system log | Debug Worker Processes That Hang or Crash |
| Workers hang while starting | Stacks of the parent and a child, and the start method | How Python Starts Worker Processes |
| It only goes wrong in a container | Python version, start method, CPU and memory limits | Size a Process Pool to a Container's CPU Limit |
| Latency rises under load | Queue lengths, waits for a pool, time per handler | Limit Concurrency, Request Rate and Queued Work |
Before you read a CPU percentage, check whether it means one core, the whole machine or a container's quota. Then compare your evidence with the time window in which requests failed.
Stuck async tasks
On Python 3.14 and later, python -m asyncio ps PID lists the pending tasks
in a running process. python -m asyncio pstree PID shows which task awaits
which. A task waiting on a queue may be healthy, or its producer may
have died. Follow what it waits for until you reach the cause.
Stuck threads
Take two or three dumps a few seconds apart. A thread that shows the same line
each time is stuck there, and the line tells you what it waits for. In a
test that hangs, faulthandler.dump_traceback_later(5.0) prints every stack
if the test is still running after five seconds. Call
faulthandler.cancel_dump_traceback_later() when the test finishes in time.
A worker process died
multiprocessing.Process.exitcode is negative when a signal killed the
process, so -9 means SIGKILL. A shell or a container tool reports the
same death as 137, which is 128 plus 9.
Check your understanding A worker's exit code is -9. Did it run out of memory? Show answer Hide answer
Perhaps. On Linux the out-of-memory killer sends SIGKILL, but so can a
supervisor, a person or a container runtime enforcing a limit. Check the
kernel log or the container's events for an out-of-memory record before you
shrink your data. An exit code of None means the process has not finished
yet.
A busy core
A sampling profiler reads the program's stack many times a second and counts
the functions it finds there. Code that shows up in many samples is where the
time goes. Python 3.15 adds one to the standard library, in the
profiling.sampling module. Python 3.15.0 came out on 9 October 2026.
python3.15 -m profiling.sampling attach PID
Its --mode cpu option leaves out time spent waiting. On Python 3.12 to
3.14, use a third-party profiler such as py-spy, and check that its release
supports your Python version. Attaching to another process needs extra
permissions on most systems, so try it in your real environment before you
need it.
Growing memory
Start tracemalloc early, take a snapshot, run the suspect workload, and
take a second snapshot. second.compare_to(first, "lineno") lists the lines
whose allocations grew. tracemalloc sees only memory that Python allocated
after it started. Also watch the process's resident memory, the memory the
operating system says the process uses, because many C libraries allocate
outside tracemalloc's view.
Prepare before the incident
- Register
faulthandlerat startup, and keep standard error. - Log a request ID with each line, and record queue lengths and pool waits.
- Try attaching a profiler in the environment where the service runs.
Going further
See the documentation for faulthandler,
tracemalloc and the
3.15 sampling profiler.