Skip to content
← All articles

A library that does not force a start method on its users

A library calling set_start_method at import breaks every downstream application in a way the application author cannot fix without vendoring you — the concurrency equivalent of logging.basicConfig() in a library.

There is a category of library bug that the application author literally cannot work around. Not “cannot work around easily” — cannot work around, short of vendoring your source and editing it.

# yourlib/__init__.py
import multiprocessing

multiprocessing.set_start_method("fork")

set_start_method is process-global and may be called once. Your import ran first. The application’s own call now raises RuntimeError: context has already been set. Every other library in the process that needed a different start method is broken. There is no override, no parameter, no environment variable. The application author’s options are: stop using your library, or fork it.

This is the concurrency equivalent of logging.basicConfig() in a library — a global decision taken by a component that had no standing to take it.

The documentation is direct about this. Libraries using multiprocessing or ProcessPoolExecutor should be designed to allow their users to provide their own multiprocessing context, and should always document if they require a specific start method.

The rules

Never call set_start_method. Not at import, not in a helper, not “defensively”. Accept a BaseContext if you genuinely need one:

def __init__(self, *, ctx: BaseContext | None = None) -> None:
    self._ctx = ctx if ctx is not None else multiprocessing.get_context()

get_context() with no argument returns whatever the application chose. That is the correct default: inherit the decision, do not make it.

Never create a pool at import time. A module-level _POOL = ProcessPoolExecutor() forks or spawns worker processes when somebody types import yourlib — in their test suite, in their linter, in their documentation build. Create it in a constructor or a context manager, where the caller controls the lifetime.

Prefer accepting an Executor. A signature that takes executor: Executor | None = None lets the caller supply a thread pool, a process pool, or — from 3.14 — an InterpreterPoolExecutor, and swap between them in one line that the type checker validates. A signature naming ThreadPoolExecutor forecloses that.

Own what you create, borrow what you are given. If you built the executor, shut it down. If it was handed to you, leave it alone — it probably belongs to a composition root that is sharing it with three other components.

Keep everything on the wire picklable and public. Anything crossing a process boundary is pickled by reference to its qualified name. A local function, a lambda, or a class defined inside another function cannot be pickled, so a callback parameter that works fine with a thread pool fails with a process pool. If your API takes a callable that may cross processes, document that it must be importable.

Never mutate process-global state outside an opt-in initializer. signal.signal, os.environ, faulthandler.enable(), warnings.filterwarnings, logging.basicConfig, sys.setrecursionlimit — all of these are the application’s to decide. If your library needs a worker process configured a particular way, expose it as an initializer the caller passes to the executor, not as something you do to their process.

💡Your library needs fork for performance — copy-on-write of a large read-only dataset. What do you ship? click to reveal

A documented requirement and a parameter. Not a set_start_method call.

Concretely: accept ctx: BaseContext | None = None. Document, in the docstring and the README, that the copy-on-write benefit requires a fork context and that the caller can pass multiprocessing.get_context("fork"). If the difference is large enough to matter, measure it and put the number in the docs so the caller can decide whether it is worth the trade.

Why the caller and not you: fork is unsafe in a process with threads, which the application may well have and you cannot know about. It is unavailable on Windows and problematic on macOS. And an application combining your library with another that needs spawn has to be able to choose. Every one of those facts is knowable at the top of the program and unknowable inside your module.

If you want to be genuinely helpful, detect the situation and warn rather than decide: if threading.active_count() > 1 and the context is fork, emit a warning explaining the risk. That gives the caller information without taking the decision away.

Why import time is the specific problem

Everything above is a special case of one rule: a module body should define things, not do things. The reason import time is uniquely bad, rather than merely early, is that the caller has no opportunity to intervene. By the time their first line of code runs, the pool exists, the signal handler is installed, the environment variable is set, the start method is fixed.

A constructor can be configured. A context manager can be scoped. An import can only be avoided by not importing you.

💡How would you prove, in CI, that your library has no import-time side effects? click to reveal

A fresh subprocess with a poisoned environment and an assertion about sys.modules:

code = (
    "import sys\n"
    "before = set(sys.modules)\n"
    "import yourlib\n"
    "assert 'sqlite3' not in sys.modules\n"
    "assert not sys.modules.get('yourlib._pool')\n"
    "import threading; assert threading.active_count() == 1\n"
)
subprocess.run([sys.executable, "-P", "-c", code], check=True)

Three assertions, three classes of side effect: nothing heavy was imported, nothing was constructed, no thread was started. Add os.environ = {} before the import to catch configuration read at import time, and run it with -P so you are testing the installed package rather than the working tree.

The static counterpart — an AST check for the specific constructs — is the problem below. The two are complementary: the subprocess test catches the effect no matter how it was written, the AST check names the line.