Debug Worker Processes That Hang or Crash

Your nightly job sends 10,000 images through a ProcessPoolExecutor. One morning it has failed with BrokenProcessPool. Another morning it is still running, with no output and no CPU use. First decide which of these happened, because each leaves different evidence:

A task raised an exception. The pool stores the exception in the task's future.
A worker process died. The pool raises BrokenProcessPool.
A process is alive but stuck. Nothing raises, so you look at where it waits.

Read the future when a task raises

The worker sends a task's exception back to the parent, and future.result() raises it there. If you never read the futures, nobody sees the error.

from concurrent.futures import ProcessPoolExecutor


def parse(n: int) -> int:
    if n == 3:
        raise ValueError(f"bad record {n}")
    return n * 2


if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        futures = [pool.submit(parse, n) for n in range(5)]
    for future in futures:
        try:
            print(future.result())
        except ValueError as error:
            print(f"task failed: {error}")

This prints 0, 2 and 4, then task failed: bad record 3, then 8. One failed task does not break the pool.

Find out what killed a worker

BrokenProcessPool means a worker ended without reporting back. The pool cannot tell what that worker was doing, so every unfinished future raises BrokenProcessPool, and so does every later submit. Results you already have are still good. Create a new pool for the tasks that remain.

These causes are common:

The operating system killed it. On Linux the out-of-memory killer is the usual cause, so check the kernel log with dmesg or your container platform's events.
It crashed inside C code, such as an extension library.
It could not load your function. A function defined inside the if __name__ == "__main__": block does not exist in a spawned worker, and the worker's stderr shows an AttributeError that names __mp_main__. Define worker functions at the top level of the module.
The pool's initializer raised an exception.

A dead worker may have written to a file or a database first. Retry only the tasks that are safe to run twice.

Make workers print their own stacks

Turn on faulthandler in a pool initializer. This works on Linux and macOS:

import faulthandler
import os
import signal
import sys
from concurrent.futures import ProcessPoolExecutor


def init_worker() -> None:
    faulthandler.enable()
    faulthandler.register(signal.SIGUSR1, all_threads=True)
    print(f"worker ready: pid={os.getpid()}", file=sys.stderr, flush=True)


def square(value: int) -> int:
    return value * value


if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2, initializer=init_worker) as pool:
        print(list(pool.map(square, [2, 3, 4])))  # [4, 9, 16]

faulthandler.enable() prints a traceback if the worker crashes inside C code. register lets you ask for one at any time. While the job is stuck, run kill -USR1 1234 with a worker PID from the log. The worker prints each thread's stack to stderr and keeps running. Without the handler, SIGUSR1 ends the process, so send it only to workers that ran this initializer.

Dump the stack of a stuck process

py-spy, installed with pip install py-spy, prints the current stack of every thread in a running Python process. It needs no setup in the process:

py-spy dump --pid 12345

Run it on the parent and on each worker. A waiting parent looks stuck when one worker is the real problem. py-spy reads another process's memory, so it needs permission. On macOS, run it with sudo. In Docker, start the container with --cap-add SYS_PTRACE.

Fix the common hangs

The parent hangs in join() after workers put results on a queue. Read every result from the queue first, then join.
A new worker hangs at once, in a program that also runs threads. With fork, it can inherit a lock that another thread held. Switch to spawn or forkserver.
At shutdown, the resource tracker warns about leaked shared memory. Call close() in every process that uses the block, and call unlink() once, in the owner.

Check your understanding A batch job sometimes ends with BrokenProcessPool, and a retry succeeds. Does that prove the workers ran out of memory? Show answer

No. It only says that a worker died. Look for an out-of-memory kill in the kernel log or container events at that moment. Then look for a crash or a startup error in the worker's stderr. If memory is the cause, run fewer workers or split the largest inputs. Recycling workers with max_tasks_per_child does not help when one task alone needs too much.

Going further

For a multiprocessing.Process you start yourself, read exitcode after join(). A negative value is the signal that ended it, so -9 is SIGKILL.
Python 3.15 adds the dead worker's PID and exit code to the BrokenProcessPool traceback.

Read more in the py-spy guide, the faulthandler reference and the concurrent.futures reference.