Skip to content

← True Parallelism and the Runtime step 12 of 18

Medium Framework

What Actually Crosses a Process Boundary

Every “multiprocessing is broken” ticket reduces to one fact:

Pickle serialises functions and classes by fully-qualified name, not by value.

pickle.dumps(worker) does not capture worker‘s bytecode. It writes down ("mymodule", "worker"). The child process then imports mymodule and looks the name up. If the lookup fails — or if the name resolves to a different object than the parent had — you get a PicklingError, an AttributeError deep in the child’s bootstrap, or, worst of all, silently wrong behaviour.

That single fact explains the entire folklore:

  • Lambdas fail — they have no qualified name to write down.
  • Closures fail — the captured cell is not part of the name.
  • Nested functions fail — outer.<locals>.inner is not importable.
  • Classes defined in a Jupyter cell or a REPL fail — same reason.
  • if __name__ == "__main__": is mandatory — because under spawn and forkserver the child re-imports your __main__ module to resolve those names, and any unguarded top-level work runs again in every child.

Live OS resources are the other half. A socket, a file descriptor, a threading.Lock, a CUDA context and a memoryview are all handles into this process’s kernel state. Pickle refuses them, and that refusal is a feature: a copied file descriptor number in another process points at something else entirely.

Your task

You are given a specimen table — a fixed set of representative objects, each reachable by a name beginning with @:

Name The object
@module_function math.sqrt
@builtin len
@partial functools.partial(math.sqrt)
@lambda a lambda
@closure a nested function that captured a local
@generator a live generator object
@lock threading.Lock()
@counter collections.Counter({"a": 1})
@bound_method a bound method of a picklable instance
@module the math module object itself
@pattern re.compile("a+")
@memoryview memoryview(b"abc")
@range range(3)

Implement:

def solve(payload: Recipe) -> list[str]:

payload is a JSON-shaped tree: dicts, lists, and leaves. Every leaf that is a string starting with @ names a specimen; any other leaf (an int, a float, True, None, an ordinary string) is itself. Return the sorted paths of every leaf that would not survive a spawn boundary.

Paths use $` for the root, `.key` to descend into a dict, and `[i]` to descend into a list. So ```python solve(payload=["@lock", {"x": "@generator"}]) # -> ["$[0]", "$[1].x"] ``` This is the audit you run on an argument list before it reachesexecutor.submit, and it is the thingmypy –strictcategorically cannot do for you. ## The typing lesson The checker **has no notion of picklability**. Both of these type-check perfectly and both explode at runtime: ```python ex.submit(takes_config, NotPicklable()) # clean under --strict ex.submit(lambda: counter) # clean under --strict ```–strictverifies the *signature* of the callable you submit. It cannot see the process boundary at all. That asymmetry — a fully-typed program with a completely unchecked seam at its most dangerous point — is the reason this track exists. The type you *can* usefully write is the recursive alias for the payload itself: ```python type Recipe = str | int | float | bool | None | list[Recipe] | dict[str, Recipe] ``` PEP 695 aliases are lazily evaluated, so a self-reference works with no quoting, andisinstance(node, dict)` narrows it cleanly on the way down.

Loading visualization…