Every other kind of Python bug hands you a traceback. Multiprocessing bugs hand you a process table. The tools you reach for by reflex — read the exception, read the log — are precisely the ones that produce nothing, because the failure happened in a process whose stderr goes nowhere and which is not going to exit.
So the skill is a different one: getting information out of a process that is not cooperating.
The triage tree
1. Dump every child’s stack, without their cooperation.
py-spy dump --pid 12345
py-spy reads another process’s memory and reconstructs its Python stack. It needs no import, no signal handler, no prior arrangement, and it works on a process that is completely wedged. Run it on the parent and on every child. This one command answers “where is it stuck” in most incidents, and it is the reason to install py-spy on production images before you need it.
On Linux you may need --nonblocking, or CAP_SYS_PTRACE / ptrace_scope relaxed. Sort that out in advance, not at 3 a.m.
2. Pre-arm the process to dump itself.
def init_worker() -> None:
import faulthandler, signal
faulthandler.enable()
faulthandler.register(signal.SIGUSR1)
Put that in the pool initializer. Now kill -USR1 <pid> makes the child print its C and Python stack to stderr — including stacks inside C extensions, which py-spy renders less usefully. faulthandler.enable() alone also converts a segfault from a silent death into a stack trace, which is the difference between “worker 3 died” and “worker 3 died in libjpeg“.
3. Read the exit code. It is a message.
Process.exitcode |
Meaning |
|---|---|
0 |
clean exit |
1 |
uncaught exception in the child |
-N |
killed by signal N |
-9 |
SIGKILL — almost always the OOM killer. Check dmesg |
-11 |
SIGSEGV — a C extension, not your Python |
-15 |
SIGTERM — something asked it to stop; was that you? |
None |
not started, or still running |
-9 deserves emphasis. A worker that vanishes with no traceback and no log, on a box with memory pressure, was killed by the kernel. Nothing in Python will tell you; dmesg | grep -i oom will.
4. Understand BrokenProcessPool.
When a worker dies uncleanly, the pool is unrecoverable. Every pending future fails, and there is no partial recovery — you cannot retry the survivors or drain what completed. The exception tells you a worker died; it does not tell you why, because the pool never saw a traceback either.
The practical consequence is architectural: if you need per-task resilience, the retry has to live above the pool, and the pool has to be cheap to recreate. Treating a ProcessPoolExecutor as a long-lived singleton makes a single OOM in a single task fatal to the whole run.
Four signature failures
Hang immediately after start(), under fork. An inherited mutex, held by a thread that no longer exists. See the previous item. py-spy will show the child stopped inside malloc, inside a logging call, or inside a BLAS entry point.
Hang at join(), only with large payloads. The feeder-thread deadlock: the child cannot exit until its queue buffer has drained into the pipe, the pipe is full, and the parent is blocked in join() instead of reading. The “only with large payloads” part is the tell — it appears when an item exceeds the ~64 KiB pipe buffer, so it passes every small-input test.
AttributeError: Can't get attribute 'f' on <module '__main__' ...>. The child re-imported __main__ and could not find the name. Either the function is not defined at module level, or there is no if __name__ == "__main__": guard. This is the friendliest error in the whole module, because it names the missing attribute.
resource_tracker: There appear to be N leaked shared_memory objects. A process that created shared memory died before unlink(), or a consumer attached without track=False and was registered as if it owned the segment. Harmless as a warning; a real leak if the killed process was the owner.
💡A batch job runs 8 workers over 40,000 images. It completes correctly most nights. Some nights it finishes with BrokenProcessPool after about 30 minutes. There is no traceback anywhere, the parent's log just stops, and re-running the same input succeeds. Where do you look, in what order, and what is the most likely cause?
click to reveal
Order matters here, because the cheapest signal is also the most likely answer.
First: dmesg on the host, filtered for OOM. A BrokenProcessPool with no traceback means a worker died without Python getting a chance to report — which is signal death, and the signal is almost always SIGKILL from the OOM killer. “Most nights it works” plus “re-running succeeds” is the signature of a memory ceiling you are near but not always over: a slightly larger image, a slightly unluckier interleaving of which workers peak at once.
Second: capture the exit code. If you are on Executor, you are not given it, which is itself a reason to reproduce with a raw Process or to log os.getpid() and reconcile against the kernel log. -9 confirms the diagnosis in one line.
Third: only now, py-spy and faulthandler. They are the right tools for a hang. This is not a hang — the pool broke and the parent raised. Reaching for the stack dumper first is the common wasted hour.
The most likely cause is not a leak in your code. It is that peak memory is roughly workers x per-task peak, and per-task peak is set by your largest input, not your average. Eight workers each decoding one 8000x6000 image at once is several gigabytes that never appear in any average. The fixes, in increasing order of effort: max_tasks_per_child to reset drift, fewer workers, sort or shard the input so the giant items do not co-occur, and a memory cgroup limit on the container so you fail predictably rather than by lottery.
A useful confirmation: log peak RSS per task with resource.getrusage(RUSAGE_SELF).ru_maxrss in the worker. If the distribution has a long right tail, that is your answer and you did not need a debugger for it.
The almost-unknown tool
import logging, multiprocessing
multiprocessing.log_to_stderr(logging.DEBUG)This turns on the module’s own internal tracing — process start-up, the feeder thread, semaphore acquisition, resource tracker activity. It is verbose and it is exactly what you want when you are trying to establish which stage a start-up is stalling in. Most people have never seen it.
What is coming
3.15 improves the diagnostic for an abruptly-terminated child to include the PID and exit code. That is a small change with a large effect on triage: today, “a process in the process pool was terminated abruptly” tells you nothing you can act on, and the first ten minutes of every such incident go into recovering the two facts the runtime already had.