Find Memory Growth with tracemalloc

Your service uses a little more memory every hour, and you want to know which line of code is responsible. tracemalloc records where Python allocates memory. Take a snapshot, run the suspect code, take another snapshot, and compare the two:

import tracemalloc

retained: list[bytearray] = []

tracemalloc.start()
try:
    before = tracemalloc.take_snapshot()
    for _ in range(100):
        retained.append(bytearray(4096))  # kept on purpose, like a leak
    after = tracemalloc.take_snapshot()
    for stat in after.compare_to(before, "lineno")[:3]:
        print(stat)
finally:
    tracemalloc.stop()

The first line of output names the bytearray line, with about 400 KiB more than before. The next lines are small allocations made by tracemalloc itself. compare_to returns one entry per source line, largest change first. Each entry's size_diff is the change in bytes since the first snapshot.

This is traced memory, memory that Python allocated while tracemalloc was running. Start tracing before the code you want to watch. Memory allocated earlier does not show up.

Read the signed difference

size_diff can be negative, when a line holds less memory than before. compare_to sorts by the size of the change, so a large drop can come before a small growth. To see only growth, filter and sort yourself:

growth = [s for s in after.compare_to(before, "lineno") if s.size_diff > 0]
growth.sort(key=lambda s: s.size_diff, reverse=True)

Pass "filename" instead of "lineno" to group by file. To see who called a shared helper, start tracing with more frames, as in tracemalloc.start(10), and group by "traceback". More frames cost more time.

The line that allocated is not the owner

A result inside a JSON decoder tells you which line allocated the memory. It does not tell you which cache or queue still holds the decoded data. Follow the callers, and look at what your code keeps.

Run the code once before the first snapshot. The first call often fills caches and imports modules, and that memory would look like growth.

Peaks are a separate question

Two snapshots show what is still held. They miss a large temporary list that Python freed before the second snapshot. tracemalloc.get_traced_memory() returns two numbers, the traced memory now and the highest it has been:

import tracemalloc

tracemalloc.start()
total = sum([n * n for n in range(100_000)])  # builds the whole list first
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(current < peak)  # True, the list was freed after sum()

tracemalloc.reset_peak() sets the peak to the current value, so you can measure one step at a time. It does not free anything.

Traced memory is not the process's memory

Your operating system usually shows the resident set size, or RSS, which is how much RAM it has given the process. RSS is usually larger than the traced total. It includes memory allocated before tracing started, memory Python freed but kept for reuse, and memory that some C libraries allocate without telling tracemalloc. So a gap between the two numbers is not a leak by itself.

Stop tracing in finally, because tracing slows your program down. Check tracemalloc.is_tracing() before you start. If someone else's session is running, tracemalloc.stop() would throw away the traces it recorded.

Going further

The tracemalloc documentation covers filters and saving snapshots to files. NumPy reports its array memory to tracemalloc, as its memory documentation describes.