Skip to content

← True Parallelism and the Runtime step 15 of 18

Hard Framework

shared_memory, the Resource Tracker, and track=

This is the only part of multiprocessing where getting it wrong leaks kernel resources that outlive your process. A crash-looping service can fill /dev/shm and take down the node, and on POSIX the segments persist until reboot.

It is also, by a wide margin, the fastest way to move bulk data:

Moving 64 MiB Time
filling a SharedMemory.buf 4.3 ms
the same 64 MiB x 5 through a Queue 110 ms

The queue figure works out to about 2.9 GiB/s of pure pickle-plus-pipe copy. For a NumPy array or a model shard, shared memory is not an optimisation, it is a different order of magnitude.

The lifecycle, precisely

  • close() drops this process’s mapping of the segment. Local. Cheap. Idempotent.
  • unlink() destroys the segment itself. Global. Must be called exactly once, by exactly one owner, after every attacher has closed.

On POSIX, spawn and forkserver start a resource tracker process whose job is to unlink anything still registered when the program ends. It is a safety net, not a guarantee — the docs are explicit:

if a process was killed by a signal there may be some leaked resources… not automatically unlinked until the next reboot.

Which is why you have seen this:

UserWarning: resource_tracker: There appear to be 1 leaked shared_memory
objects to clean up at shutdown

That is cpython issue #82300, open since 2019. The most common cause is benign: a consumer process attached to a segment it does not own, and the tracker registered it anyway. Since 3.13 there is a sanctioned fix — SharedMemory(name=..., track=False) — which says “I am attaching, I am not the owner, do not register me”.

Platform notes worth carrying: Windows reference-counts handles, ignores track, and has no unlink() at all. And ShareableList has a documented nul-byte-stripping bug (gh-106939) — "a\x00b" does not round-trip.

Your task

Wrap the lifecycle so it cannot be got wrong.

class SharedArray:
    @classmethod
    def create(cls, size: int) -> Self: ...      # owner: will unlink
    @classmethod
    def attach(cls, name: str) -> Self: ...      # track=False, never unlinks
    @property
    def buf(self) -> memoryview[int]: ...        # raises after close
    def close(self) -> None: ...
    def unlink(self) -> None: ...                # at most once, owner only
    def __enter__(self) -> Self: ...
    def __exit__(self, ...) -> None: ...         # close, then unlink
def solve(*, size: int, data: str, offset: int) -> tuple[bytes, bool, bool]:

Create a segment, write data at offset, attach a second handle by name, read the bytes back through it, and let both context managers exit. Then return: the bytes seen through the attached handle, whether buf raises after close, and whether a second unlink() is safe.

The composition is the test. If attach unlinked on exit, the owner’s unlink would then fail and the third flag goes false. If unlink went through a freshly-constructed handle rather than the original object, the resource tracker would never be told and you would get the leak warning above.

Two traps this problem is defending against

buf returning None after close. Upstream, SharedMemory.buf is typed memoryview[int] | None, and a public API that propagates that None pushes a check onto every caller forever. Raise instead: RuntimeError at the moment of misuse beats an AttributeError three frames away.

Keeping a NumPy view alive past close() segfaults. It is not an exception you can catch — the mapping is gone and the array still points at it. Any np.ndarray(..., buffer=shm.buf) must be dropped, or copied, before the handle closes. That is the strongest practical reason to own the lifecycle in a context manager rather than by convention.

Loading visualization…