Skip to content

← Parallel Python with Process Pools step 18 of 24

Medium Framework

Fix Multiprocessing Type Hints That Fail at Runtime

With spawn or forkserver, each worker process imports your module again before it runs a task. That import runs every def in the module, annotations included. Here is a worker function that mypy accepts:

import multiprocessing


def drain(queue: multiprocessing.Queue[int]) -> None: ...

multiprocessing.Queue is a function that makes a queue, not the queue class, so Python raises TypeError when it evaluates multiprocessing.Queue[int]. Python 3.13 and older evaluate annotations when the def runs, so the import fails. Python 3.14 and later wait until something reads them, such as typing.get_type_hints or a library like pydantic. The error then appears at that later read, which can happen inside a worker.

Submodules cause a second problem. import multiprocessing does not load multiprocessing.managers or the other submodules in the table below. An annotation that names one can work in the parent, where some other import loaded it, and raise AttributeError in a fresh worker. Import each submodule you name.

Annotate with the class, create through the context

Keep creating objects through a context, as in multiprocessing.get_context("spawn").Queue(). In annotations, name the class from its submodule.

What you create Class for annotations
Lock, event, semaphore multiprocessing.synchronize.Lock, Event, Semaphore
Queue, simple queue multiprocessing.queues.Queue, SimpleQueue
Process pool multiprocessing.pool.Pool
Manager dictionary multiprocessing.managers.DictProxy
Pipe end multiprocessing.connection.Connection
from multiprocessing.queues import Queue
from typing import get_type_hints


def enqueue(queue: Queue[int], value: int) -> None: ...


print(get_type_hints(enqueue)["queue"])  # multiprocessing.queues.Queue[int]

Connection[bytes, bytes] raises TypeError at runtime, even though type-checker stubs describe the class as generic. Use bare Connection.

What to write

Fix the imports and the obj annotation of the eight functions in the starter. Keep the element types Queue[int], SimpleQueue[bytes] and DictProxy[str, int], and use bare Connection. typing.get_type_hints must work on every function in REGISTRY, and the file must pass mypy --strict with no # type: ignore.

Then write solve(*, functions). For each name in functions:

  1. Look the function up in REGISTRY.
  2. Call typing.get_type_hints on it and take the "obj" entry.
  3. If typing.get_origin returns a class, use that class. Otherwise use the hint itself. Queue[int] has the origin Queue, and a plain class has none.
  4. Store that class's __module__ under the requested name.

For example, solve(functions=["takes_lock"]) returns {"takes_lock": "multiprocessing.synchronize"}.

  • Read each module from the annotation. Don't hard-code the strings or create multiprocessing objects in solve.
  • For an empty list, return {}. A repeated name gives one entry.
  • An unknown name raises KeyError.
  • Leave the input list and REGISTRY unchanged.

The tests ask about subsets of the registry, but all eight annotations must be right.