Speed Up CPU-Heavy Work with a Process Pool

In the concurrency track, a thread pool made waiting faster but did nothing for code that computes. A process pool fixes that. Save this script as speedup.py and run it with python speedup.py:

import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor


def count_primes(limit: int) -> int:
    '''Count the primes below limit, the slow way, in pure Python.'''
    count = 0
    for n in range(2, limit):
        if all(n % d for d in range(2, int(n**0.5) + 1)):
            count += 1
    return count


def main() -> None:
    jobs = [500_000] * 4

    start = time.perf_counter()
    plain = [count_primes(n) for n in jobs]
    print(f"plain loop:   {time.perf_counter() - start:.1f}s")

    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=4) as pool:
        threaded = list(pool.map(count_primes, jobs))
    print(f"thread pool:  {time.perf_counter() - start:.1f}s")

    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=4) as pool:
        processed = list(pool.map(count_primes, jobs))
    print(f"process pool: {time.perf_counter() - start:.1f}s")

    assert plain == threaded == processed


if __name__ == "__main__":
    main()

On a 10-core laptop with Python 3.15, it printed:

plain loop:   1.8s
thread pool:  1.8s
process pool: 0.6s

Your numbers will differ, but the pattern holds on any machine with several cores. The thread pool is no faster than the loop. The process pool is about three times faster with four workers.

Why processes help

Each worker in a process pool is a separate Python process, with its own interpreter and its own GIL. Four processes can run Python code on four cores at the same time. Four threads in one process take turns on one GIL.

ProcessPoolExecutor has the same methods as ThreadPoolExecutor. map keeps the input order, submit returns a Future, and the with block waits for every job.

What a process pool needs from your code

• The function must be importable. Pickle sends a function as its module and name, not as its code, and the worker looks it up by that name. Define the function at the top level of a module, not inside another function.
• The pool must start under if __name__ == "__main__":. Depending on how Python starts workers, a worker can run your module's top-level code again. Without the guard, each worker would try to start its own pool.
• Arguments and results must be picklable. Pickling is how Python turns an object into bytes to send it to another process. Numbers, strings, lists, dicts and top-level functions pickle. Lambdas, nested functions, open files and locks do not.

What a process pool costs

Starting a process takes far longer than starting a thread, and Python copies every argument and result between processes. A process pool is faster only when each job does a lot of computing compared with the data it sends. Thousands of tiny jobs can run slower than a plain loop, and a later article shows how to batch them.

Why the exercises here simulate

The runtime that grades your exercises runs your code in a way that cannot send your own functions to a worker process. The exercises in this track therefore check, plan or simulate what a process pool does, and some run provided code. Run this article's script on your own machine to see a real pool at work.

Going further

Python's free-threaded build can run threads on several cores without a process pool. A later article in this track covers it.