We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
← True Parallelism and the Runtime step 15 of 18
shared_memory, the Resource Tracker, and track=
This is the only part of multiprocessing where getting it wrong leaks
kernel resources that outlive your process. A crash-looping service can
fill /dev/shm and take down the node, and on POSIX the segments persist
until reboot.
It is also, by a wide margin, the fastest way to move bulk data:
| Moving 64 MiB | Time |
|---|---|
filling a SharedMemory.buf |
4.3 ms |
the same 64 MiB x 5 through a Queue |
110 ms |
The queue figure works out to about 2.9 GiB/s of pure pickle-plus-pipe copy. For a NumPy array or a model shard, shared memory is not an optimisation, it is a different order of magnitude.
The lifecycle, precisely
-
close()drops this process’s mapping of the segment. Local. Cheap. Idempotent. -
unlink()destroys the segment itself. Global. Must be called exactly once, by exactly one owner, after every attacher has closed.
On POSIX, spawn and forkserver start a resource tracker process whose
job is to unlink anything still registered when the program ends. It is a
safety net, not a guarantee — the docs are explicit:
if a process was killed by a signal there may be some leaked resources… not automatically unlinked until the next reboot.
Which is why you have seen this:
UserWarning: resource_tracker: There appear to be 1 leaked shared_memory
objects to clean up at shutdown
That is cpython issue #82300, open since 2019. The most common cause is
benign: a consumer process attached to a segment it does not own, and the
tracker registered it anyway. Since 3.13 there is a sanctioned fix —
SharedMemory(name=..., track=False) — which says “I am attaching, I am not
the owner, do not register me”.
Platform notes worth carrying: Windows reference-counts handles, ignores
track, and has no unlink() at all. And ShareableList has a documented
nul-byte-stripping bug (gh-106939) — "a\x00b" does not round-trip.
Your task
Wrap the lifecycle so it cannot be got wrong.
class SharedArray:
@classmethod
def create(cls, size: int) -> Self: ... # owner: will unlink
@classmethod
def attach(cls, name: str) -> Self: ... # track=False, never unlinks
@property
def buf(self) -> memoryview[int]: ... # raises after close
def close(self) -> None: ...
def unlink(self) -> None: ... # at most once, owner only
def __enter__(self) -> Self: ...
def __exit__(self, ...) -> None: ... # close, then unlink
def solve(*, size: int, data: str, offset: int) -> tuple[bytes, bool, bool]:
Create a segment, write data at offset, attach a second handle by name,
read the bytes back through it, and let both context managers exit. Then
return: the bytes seen through the attached handle, whether buf raises after
close, and whether a second unlink() is safe.
The composition is the test. If attach unlinked on exit, the owner’s unlink
would then fail and the third flag goes false. If unlink went through a
freshly-constructed handle rather than the original object, the resource
tracker would never be told and you would get the leak warning above.
Two traps this problem is defending against
buf returning None after close. Upstream, SharedMemory.buf is typed
memoryview[int] | None, and a public API that propagates that None pushes
a check onto every caller forever. Raise instead: RuntimeError at the moment
of misuse beats an AttributeError three frames away.
Keeping a NumPy view alive past close() segfaults. It is not an
exception you can catch — the mapping is gone and the array still points at
it. Any np.ndarray(..., buffer=shm.buf) must be dropped, or copied, before
the handle closes. That is the strongest practical reason to own the lifecycle
in a context manager rather than by convention.
Loading visualization…
Stuck?
Python reference solution
Sign in to attempt this problem and reveal the reference solution.