Zero-Copy IPC With Pickle Protocol 5 Out-of-Band Buffers

At the measured 2.9 GiB/s that a multiprocessing.Queue sustains, a 1 GiB batch costs roughly 350 ms of pure overhead per hop — pickling into a buffer, writing it down a pipe, reading it back, unpickling into a new allocation. Knowing whether your channel can avoid that is the difference between a pipeline that scales and one that saturates on memcpy.

What PEP 574 added

Protocol 5 introduced out-of-band buffers. A type’s __reduce_ex__ may wrap its large payload in a pickle.PickleBuffer when protocol >= 5. The sender passes buffer_callback=, which diverts those buffers out of the byte stream; the receiver passes buffers= to put them back. The stream itself carries only the metadata.

bufs: list[pickle.PickleBuffer] = []
stream = pickle.dumps(payload, protocol=5, buffer_callback=bufs.append)
restored = pickle.loads(stream, buffers=bufs)

For a 64 KiB payload, len(stream) drops from ~65,000 bytes to about 30. And the reconstructed object is backed by the same memory — mutate the original and the “copy” changes, because there is no copy.

3.14 raised pickle.DEFAULT_PROTOCOL to 5, so multiprocessing now gets protocol-5 framing by default.

The part that disappoints people

multiprocessing.Queue does not opt in. ForkingPickler is used without a buffer_callback, so protocol 5 or not, you still get in-band copies. The default protocol changing did not make your queue zero-copy.

To actually benefit you must either register a custom reducer that hands off a shared_memory name instead of the bytes, or use a library that already did the work — torch.multiprocessing, Arrow, Ray. This is the single most common misconception about PEP 574: the machinery exists, and the transport you are using has not enabled it.

Your task

Build the codec by hand, so you can tell at a glance whether a channel is out-of-band-capable.

def pack(tag: str, payload: bytearray, protocol: int) -> tuple[bytes, list[PickleBuffer]]: ...
def unpack(stream: bytes, buffers: list[PickleBuffer]) -> tuple[str, memoryview[int]]: ...

At protocol 5 or above, wrap the payload in a PickleBuffer and collect the out-of-band buffers. Below 5, fall back to an in-band bytes copy — the frame must still round-trip correctly, just without the saving.

def solve(
    *, tag: str, pattern: str, repeat: int, protocol: int
) -> tuple[str, bool, int, bool, int, str]:

Build bytearray(pattern.encode() * repeat), pack it, unpack it, and report: the tag that came back, whether the stream is under 1 KiB, how many out-of-band buffers there were, whether the result shares memory with the source, the payload length, and the first 16 hex characters of its SHA-256.

The zero-copy flag is the one that teaches. memoryview(restored).obj is source is literally True at protocol 5 and literally False at protocol 4 — same 64 KiB payload, same API, one memcpy of difference.

Why the payload is a bytearray

PickleBuffer distinguishes writable from read-only buffers, and the distinction survives the round trip. A writable bytearray comes back as a writable view onto the original allocation; a read-only bytes cannot, since the unpickler would have to promise mutability it does not have. When you design your own __reduce_ex__, deciding which of the two you are handing out is a real API decision, not an implementation detail.

Where to take this next

The natural extension is a reducer registered on ForkingPickler that serialises a large bytearray as a shared-memory name plus size rather than its contents. The pickle stream for a 64 MiB payload then fits comfortably under 1 KB, and the receiving process attaches instead of copying — which is precisely the technique torch.multiprocessing uses to move tensors between DataLoader workers.