Write a Library That Leaves Pools and Start Methods to the Caller

You publish a small library that runs a function over a list in a process pool. So that it behaves the same on every platform, the package's __init__.py picks the start method:

# Don't do this in a library. It runs as soon as someone imports the package.
import multiprocessing

multiprocessing.set_start_method("spawn")

A program can set the start method only once. If the application that imports your library also calls set_start_method, the second call raises RuntimeError: context has already been set. If the application wanted forkserver, your import has already chosen for it.

Importing a library should change nothing outside the library. The application owns its processes, so let it make these choices.

Take the pool as an argument

A function that needs workers can take an executor from the caller. The caller creates it, and the caller shuts it down. If the caller passes nothing, the function makes its own pool and closes it before it returns. Save this as a file and run it:

from collections.abc import Callable, Iterable
from concurrent.futures import Executor, ProcessPoolExecutor, ThreadPoolExecutor
import multiprocessing


def map_values[T, R](
    fn: Callable[[T], R], values: Iterable[T], *, executor: Executor | None = None
) -> list[R]:
    if executor is not None:
        return list(executor.map(fn, values))
    with ThreadPoolExecutor() as owned:
        return list(owned.map(fn, values))


def square(value: int) -> int:
    return value * value


if __name__ == "__main__":
    assert map_values(square, [2, 3]) == [4, 9]
    context = multiprocessing.get_context("spawn")
    with ProcessPoolExecutor(mp_context=context, max_workers=2) as shared:
        assert map_values(square, [2, 3], executor=shared) == [4, 9]
        assert shared.submit(square, 4).result() == 16

map_values closes the thread pool it creates. It leaves a pool that the caller passed in open, even when a task raises, so the caller can keep using it, as the last submit does.

Take the start method as an argument too

If your library creates process pools itself, accept a context. Add a parameter mp_context: BaseContext | None = None, import BaseContext from multiprocessing.context, and pass the value on to ProcessPoolExecutor(mp_context=mp_context). None gives the application's default.

Don't look up the default with multiprocessing.get_context(). Called with no argument, it sets the default if nothing has set it yet, so a later set_start_method call in the application raises RuntimeError.

Check your understanding Your library loads a large dataset once, and with fork every worker would start with the dataset already in memory. Should the library call set_start_method("fork")? Show answer

No. Accept a context, and document that fork makes the library faster, with a measurement. fork can deadlock in a program that runs threads, and only the application knows which threads it runs. Checking threading.active_count() does not settle it, because threads that C libraries start do not appear in that count. Keep a path that works without fork, which Windows does not have.

Thread pools and process pools are not interchangeable

Executor gives both kinds of pool the same methods, but a task that works in one can fail in the other. A thread pool runs the task in your process, so the task can use any object, including a lambda or an open connection. A process pool pickles the function, its arguments and its result. So a library that accepts any Executor must pass only top-level functions and picklable data. Test with each kind of pool you support.

Keep setup out of module bodies

Python runs a module's top-level code when it imports the module, so don't create a pool there. No caller owns a pool made at import, so no caller shuts it down.

The same goes for calls that change the whole process:

logging.basicConfig() sets up the root logger, which belongs to the application.
Writing to os.environ changes the environment for every library and every child process.
signal.signal() replaces the application's signal handler.

Offer a function the application can call instead. If each worker needs setup, provide an initializer that the caller passes to its pool.

Test what importing changes

Import the library in a fresh interpreter, and compare the state before and after:

import subprocess
import sys

code = '''
import logging
import multiprocessing
import os

before_method = multiprocessing.get_start_method(allow_none=True)
before_environment = dict(os.environ)
before_handlers = tuple(logging.getLogger().handlers)

import yourlib

assert multiprocessing.get_start_method(allow_none=True) == before_method
assert dict(os.environ) == before_environment
assert tuple(logging.getLogger().handlers) == before_handlers
'''
subprocess.run([sys.executable, "-I", "-c", code], check=True, timeout=15)

Replace yourlib with your package's name, and install the package into the test environment first. The -I flag leaves the current directory off the import path, so the test imports the installed package, not your source folder.

The test catches lasting changes to the start method, the environment and the root logger. It cannot see a file the import wrote or a network call it made, so add a check for each thing your library promises not to do. The exercise below reads a library's source instead, and flags these calls without running it.

Going further

set_start_method(..., force=True) replaces a default that is already set. Queues and locks created before the change still belong to the old start method, and using them with the new one can fail.
Python 3.14's InterpreterPoolExecutor also pickles tasks and results.
The multiprocessing documentation explains contexts and start methods.