Keep Forked Workers Sharing Memory with gc.freeze

A web server loads a large lookup table, then forks eight worker processes. Right after the fork, the workers share the parent's memory, so the table takes up RAM only once. An hour later, each worker holds its own copy of much of it, although no code ever changes the table.

This article assumes the fork start method from T10's How Python Starts Worker Processes.

Reading an object can copy its memory

After fork, the parent and the worker use copy-on-write. Both read the same physical page of memory until one of them writes to it. Then the operating system gives the writer its own copy of that page.

In CPython, using an object writes to it. Each new reference to an object changes its reference count, a number stored inside the object. When the garbage collector runs, it also writes bookkeeping into each object it examines. Each of those writes can copy a page that your code only reads.

You cannot stop reference counting, but you can keep the collector away from the objects the parent built.

Freeze the parent's objects before you fork

gc.freeze() moves every object the collector tracks into a permanent generation that later collections skip. The collector then leaves those objects alone in the workers. The gc documentation gives this order:

1. Call gc.disable() early in the parent. Collections then cannot leave freed gaps between the long-lived objects while you build them.
2. Call gc.freeze() right before you fork.
3. Call gc.enable() early in each worker.

Freezing does not make objects read-only, and it does not stop reference counting. The collector still handles objects created after the freeze.

The cost is that a frozen cycle stays in memory after it becomes garbage, because the collector no longer looks at it. gc.unfreeze() moves the frozen objects back into the oldest generation, where the next full collection can free them:

import gc
import weakref


class Node:
    def __init__(self) -> None:
        self.peer: Node | None = None


was_enabled = gc.isenabled()
gc.disable()
try:
    node = Node()
    node.peer = node
    observed = weakref.ref(node)
    gc.freeze()
    del node

    gc.collect()
    assert observed() is not None  # The frozen cycle remains alive.

    gc.unfreeze()
    gc.collect()
    assert observed() is None
finally:
    gc.unfreeze()
    if was_enabled:
        gc.enable()

Run this as its own script. gc.unfreeze() affects every frozen object in the process, including any that another library froze.

Check the start method first

Freezing helps only with fork. With spawn, each worker is a fresh Python that shares nothing with the parent. With forkserver, the Linux default from Python 3.14, each worker is a copy of the fork server, not of your parent. Freezing in the parent does nothing for those workers. And never fork a program that has threads. The T10 article shows the deadlock.

Measure private memory, not summed RSS

RSS, the RAM the operating system has given a process, counts a shared page in every process that uses it. So adding up eight workers' RSS counts the shared table eight times. On Linux, /proc/<pid>/smaps_rollup reports PSS, which divides each shared page among the processes that share it. It also reports the private pages that each process owns. Compare those numbers with and without freezing, after real requests, and watch startup time and latency too.

Check your understanding Would loading the table separately in each worker avoid the problem? Show answer Hide answer

It avoids the copying, because the workers share nothing. But each worker then holds a full copy from the start, and the loading runs eight times. That trade can be right when each worker needs to change its own copy. Measure both layouts before you choose.

Turning the collector off is not a fix

gc.disable() stops automatic collections only. Reference counting still frees most objects, gc.collect() still runs when you call it, and garbage cycles pile up until something collects them. If one phase must run without automatic collections, put the previous setting back afterwards, instead of always turning collection on:

import gc
from collections.abc import Iterator
from contextlib import contextmanager


@contextmanager
def without_automatic_gc() -> Iterator[None]:
    was_enabled = gc.isenabled()
    gc.disable()
    try:
        yield
    finally:
        if was_enabled:
            gc.enable()


with without_automatic_gc():
    assert not gc.isenabled()
    result = sum(range(100))
assert result == 4950

The setting belongs to the whole interpreter. So this helper is unsafe when several threads each assume they own it.

Check your understanding Should you raise the collector's thresholds instead? Show answer Hide answer

First check that collections cause the problem. Record how often they run and how long they take, with gc.callbacks, next to latency and memory over a realistic run. Higher thresholds make collections rarer, but each one can take longer, and garbage cycles stay in memory longer. Test any change against a baseline on the exact Python version you deploy.

Going further

The gc.freeze documentation gives the pre-fork recipe. The Linux kernel's process memory reports describe PSS and the private-page counts.