Skip to content

← Parallel Python with Process Pools step 14 of 24

Hard Framework

Share Memory Between Processes and Clean It Up

Your pool's workers all need the same 200 MB array. Pass it as an argument and every worker gets its own pickled copy. With shared memory, one process creates a named block of memory, and the others attach to it by name and read the same bytes.

from multiprocessing import shared_memory

block = shared_memory.SharedMemory(create=True, size=1024)
block.buf[0:5] = b"hello"
reader = shared_memory.SharedMemory(name=block.name, track=False)
print(bytes(reader.buf[0:5]))  # b'hello'

reader.close()
block.close()
block.unlink()

Each SharedMemory object is a handle on the block, and its buf is a memoryview of the block's bytes. The grader cannot start processes, so here both handles live in one process. The cleanup rules are the same.

Close every handle, unlink once

  • close() ends one handle's access. Call it on every handle.
  • unlink() deletes the block. Only the owner, the process that created the block, calls it. The owner waits until the readers finish.

Python's resource tracker is a helper process. When a program exits, the tracker deletes every block the program created or attached to and did not unlink. A reader started as a separate program has its own tracker, and that tracker deletes the owner's block when the reader exits. So a reader attaches with track=False.

A slice of buf keeps the block in use. While one exists, close() raises BufferError and leaves the handle's buf set to None. Release the slice and close again.

What to write

Fill in SharedArray, a wrapper around one handle that knows whether it owns the block.

Member What it does
create(size) Creates a block and returns an owner.
attach(name) Attaches with track=False and returns a reader.
name The block's name, even after cleanup.
buf Returns the handle's buf. Raises RuntimeError after a successful close, or if the handle's buf is None.
close() Closes the handle. After one success, later calls do nothing.
unlink() An owner unlinks the block, and after one success later calls do nothing. A reader does nothing.
__enter__() Returns the wrapper.
__exit__(...) Calls close(), then unlink() in a finally clause. Never suppresses an exception.

Clean up through the handle you hold, not a second one. Record a close or an unlink only after the call returns. If it raises, let the error propagate so a later call can try again. That includes a FileNotFoundError if something else already deleted the block.

Then write solve(*, size, data, offset). Open an owner with with SharedArray.create(size) as owner and write the UTF-8 bytes of data at offset. Inside that with statement, open with SharedArray.attach(owner.name) as reader and read the same range. Return (bytes_seen, buf_raises_after_close, double_unlink_is_safe).

  • bytes_seen is a bytes copy of what the reader saw.
  • buf_raises_after_close is whether owner.buf raises RuntimeError after both with statements end.
  • double_unlink_is_safe is whether a second owner.unlink() returns without raising.

Offsets count bytes, not characters, and the payload always fits. data can be empty. For size=8, data="é" and offset=6, the two-byte payload fills the end of the block.