Pool vs ProcessPoolExecutor, and the chunksize Trap

Two APIs do the same job, and people benchmark them against each other and get a surprising answer. The answer is almost always a default, not a design.

The chunksize trap

# concurrent.futures.Executor.map
def map(self, fn, *iterables, timeout=None, chunksize=1): ...

# multiprocessing.pool.Pool.map — chunksize=None, computed
chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
if extra:
    chunksize += 1

Executor.map sends one item per IPC round trip unless you say otherwise. Pool.map splits the input into roughly 4 x workers chunks.

For a million tiny tasks on 8 workers, that is one million pickle-and-pipe round trips against about thirty-two. Each round trip is on the order of tens of microseconds; the work itself might be a microsecond. You are measuring the transport, and the transport is a hundred times more expensive in one case than the other.

This is the single most common reason for the claim “ProcessPoolExecutor is slower than Pool“. It is not. It has a different default, chosen because Executor.map also has to work for ThreadPoolExecutor, where chunking buys nothing and costs latency.

# The fix is one argument.
ex.map(score, rows, chunksize=max(1, len(rows) // (workers * 4)))

The rule of thumb: if per-item work is under about a millisecond, you must chunk. If it is over a hundred milliseconds, chunking costs you load balancing and you should not.

💡You replace Pool.map with ProcessPoolExecutor.map over 2,000,000 rows and wall time goes from 40 s to 25 minutes. You add chunksize and it drops to 38 s. Your colleague suggests just setting chunksize to len(rows) divided by the worker count — one chunk per worker, minimum overhead. Why is that usually the wrong number? click to reveal

Because it destroys load balancing, and load balancing is what a pool is for.

With exactly one chunk per worker, every worker gets its share of the work up front and there is no rebalancing afterwards. If the rows are uniform, fine. If they are not — and real data is not; one row has a 50 MB payload, one customer has 400x the transactions — then one worker finishes in 30 s, seven workers finish in 5 s, and your wall time is 30 s on eight cores. The tail dominates and the other seven cores idle.

4 x workers chunks is the stdlib’s compromise: enough chunks that a slow one only delays you by a quarter of a worker’s share, few enough that IPC stays negligible. That is where the * 4 in divmod(len(iterable), len(self._pool) * 4) comes from, and it is a reasonable default to copy rather than to re-derive.

There is a second reason too. One chunk per worker means the whole chunk is pickled, sent, and held in the worker’s memory at once. Eight chunks of 250,000 rows is eight large allocations that all exist simultaneously; the peak memory profile is completely different from a stream of small ones. When people report that switching to chunking caused an OOM, this is what happened.

What each API has that the other does not

Pool has, and Executor does not:

imap / imap_unordered — lazily consume the input and yield results as they arrive. This is the one that actually matters: it is how you process a stream larger than memory, and Executor.map materialises all futures immediately. (3.14 finally adds map(buffersize=) to close this gap.)
starmap — for functions taking multiple arguments, without a lambda you cannot pickle anyway.
apply_async(callback=..., error_callback=...) — a completion-callback style rather than a future style.
Automatic chunking, as above.

Executor has, and Pool does not:

A uniform Future API shared with ThreadPoolExecutor and — since 3.14 — InterpreterPoolExecutor. Switching substrate is a one-line change.
as_completed and wait, which let you react to whichever result lands first.
Real per-call typing. ex.submit(work, "x") where work takes an int is a type error; Pool.apply_async(work, ("x",)) is not. This is the argument that usually settles it.
shutdown(cancel_futures=True).
Defined BrokenProcessPool semantics.
3.14: terminate_workers(), kill_workers(), map(buffersize=).

Both: initializer / initargs, worker recycling (maxtasksperchild / max_tasks_per_child), and — since 3.13 — defaults derived from os.process_cpu_count() rather than os.cpu_count().

One platform quirk to remember: ProcessPoolExecutor caps max_workers at 61 on Windows, because of a WaitForMultipleObjects handle limit.

Guidance

Default to ProcessPoolExecutor in typed application code. The typing is genuinely better, the API is portable across all three substrates, and the shutdown story is defined rather than folklore.

Reach for Pool when you need streaming imap_unordered over an input that does not fit in memory, or when automatic chunking over a large uniform workload is exactly what you want and you would rather not compute it.

Two rules that are not optional

Never call an Executor or Future method from inside a submitted callable. Submitting from a worker, or waiting on another future from a worker, is a documented deadlock — not a race, not “usually fine”. The pool has a fixed number of workers; a worker blocked waiting for work that can only be done by a worker is a cycle.

Exceptions surface at different times, and this catches people.

results = ex.map(parse, rows)     # returns immediately; nothing has raised
for r in results:                 # <- the exception comes out HERE
    ...

results = pool.map(parse, rows)   # <- the exception comes out HERE

Executor.map returns a lazy iterator, so a try/except wrapped around the map call itself catches nothing. Pool.map is eager and raises from the call. If you are porting between them, the try block has to move — and if it does not, your error handling silently stops running.