Batch Small Jobs in Pool and ProcessPoolExecutor

You have 100,000 small jobs, and each one takes a few microseconds of Python. You hand them to a process pool with map, and the program runs eight times slower than a plain loop. Save this script as chunks.py and run it:

import time
from concurrent.futures import ProcessPoolExecutor


def score(n: int) -> int:
    return sum(i * i for i in range(n % 1_000))


def main() -> None:
    rows = list(range(100_000))

    start = time.perf_counter()
    expected = [score(n) for n in rows]
    print(f"plain loop:      {time.perf_counter() - start:.2f}s")

    for size in (1, 1_000):
        start = time.perf_counter()
        with ProcessPoolExecutor(max_workers=4) as pool:
            results = list(pool.map(score, rows, chunksize=size))
        print(f"chunksize={size:<5}: {time.perf_counter() - start:.2f}s")
        assert results == expected


if __name__ == "__main__":
    main()

On a 10-core laptop with Python 3.15, it printed:

plain loop:      0.70s
chunksize=1    : 5.70s
chunksize=1000 : 0.27s

Every message to a worker costs time

With chunksize=1, the pool sends each job on its own. For every job, the pool pickles the argument and sends it to a worker, and the worker pickles the result and sends it back. In this run the work took about 7 microseconds per job, and the pool spent about 57 microseconds per job, mostly on those messages.

With chunksize=1000, the pool sends 1,000 arguments in one message. The worker runs score on each of them and sends the 1,000 results back together. That is 100 round trips instead of 100,000. The results still come back in input order.

ProcessPoolExecutor.map uses chunksize=1 unless you say otherwise. Set it yourself whenever the jobs are small.

Pick a size that keeps every worker busy

Bigger chunks mean fewer messages, but a worker that takes a chunk runs all of it. If one chunk holds the slow jobs, its worker keeps going after the others have finished and sit idle. A big chunk also delays the first result and holds more data in memory at once.

So aim for several chunks per worker. multiprocessing.Pool does this for you when you leave chunksize out of Pool.map. It aims for about four chunks per worker:

chunksize = math.ceil(len(items) / (workers * 4))

For 100,000 items and 4 workers, that is 6,250 items per chunk, or 16 chunks. Start there, then time two or three sizes on your real input.

Check your understanding Why not make exactly one chunk per worker? Show answer

Because chunks rarely take the same time. With one chunk each, a worker that finishes early has nothing left to take, so the worker with the slowest chunk decides the total time. With several chunks per worker, a worker that finishes early takes the next chunk. A single slow job can still hold up the end, so time a few sizes on real input.

Pool and ProcessPoolExecutor differ in defaults and cleanup

Both run your function in worker processes, and both map methods return results in input order. These are the differences that cause bugs:

	`multiprocessing.Pool`	`ProcessPoolExecutor`
`map` chunk size if you leave it out	About four chunks per worker	1
`map` returns	A list, after every job has finished	An iterator that yields results in order
A job raises an exception	`map` raises it	The iterator raises it when it reaches that result
Leaving the `with` block	Calls `terminate()`, which stops the workers at once	Waits for every submitted job to finish
Other ways to run jobs	`imap`, `imap_unordered`, `starmap`, `apply_async`	`submit`, which returns a `Future`

ProcessPoolExecutor has the same methods as the ThreadPoolExecutor you used in the concurrency track, and as_completed works on its futures too. Choose it unless you need one of Pool's extra methods.

Common mistakes

Reading Pool results after the with block. imap and apply_async return before the work finishes. When the block ends, terminate() stops the workers, and reading the results can then wait forever:

with Pool(processes=2) as pool:
    lazy = pool.imap(score, rows)
results = list(lazy)  # hangs, because the workers are gone

Read the results inside the block. If you want the queued work to finish before the block ends, call pool.close() and then pool.join() inside it.

Catching executor errors in the wrong place. A job's exception reaches you when your loop reads that job's result, not when you call executor.map. Put the try around the loop that reads the results.

Going further

Since Python 3.14, ProcessPoolExecutor.map takes a buffersize argument. It limits how many chunks the pool submits ahead of the results you have read. Without it, map reads the whole input at once.
Pool.map turns an input without a length, such as a generator, into a list before it starts.
Don't call executor or future methods from inside a function that a ProcessPoolExecutor runs. The documentation warns that this can deadlock.

See the executor map documentation and the Pool documentation.