Your web service has used more memory every day since Monday, and some
requests take seconds. cProfile can't help much. It has to start with the
program, it slows down every call, and it measures time, not memory. Four
free tools from outside the standard library fill those gaps. Install each
one with pip.
| You want to know | Try |
|---|---|
| What a running process is doing, without restarting it | py-spy |
| Where memory is allocated, including inside C extensions | Memray |
| How time and memory split between Python and native code | Scalene |
| Which lines inside one function are slow | line_profiler |
See what a running process is doing with py-spy
py-spy is a sampling profiler. It reads a running program's call stack from outside, many times a second, and counts the functions it sees. It does not hook every call, so it barely slows the program down, and it can attach without a restart:
py-spy dump --pid 12345 # print each thread's stack once
py-spy top --pid 12345 # a live view, like the top command
py-spy record -o profile.svg --pid 12345 # save a flame graph
Reading another process's memory needs permission. On macOS that means
sudo, and Linux servers and containers often block it too. Try it in your
production setup before you need it in an incident.
Find where memory comes from with Memray
Memray records every allocation from the moment it starts, in Python code and in C extensions. It runs on Linux and macOS:
python -m memray run --native -o capture.bin app.py
python -m memray summary capture.bin
python -m memray flamegraph capture.bin
--native adds the C functions to each stack. An allocation record tells
you where memory came from, not what still holds it. To find out what keeps
an object alive, look at its references, as earlier in this track.
Split Python and native time with Scalene
Scalene profiles CPU time and memory together, and splits each between Python code and native code:
python -m scalene run app.py # profile, and save scalene-profile.json
python -m scalene view --cli # print the saved profile in the terminal
A large native share doesn't mean the code is already as fast as it can be. Your program may call an expensive library function more often than it needs to.
Time single lines with line_profiler
Once you know which function is slow, line_profiler times each of its lines:
from line_profiler import LineProfiler
def parse_batch(rows: list[bytes]) -> list[int]:
values: list[int] = []
for row in rows:
values.append(int(row))
return values
profiler = LineProfiler()
profiler.add_function(parse_batch)
result = profiler.runcall(parse_batch, [b"10", b"20", b"30"])
assert result == [10, 20, 30]
profiler.print_stats()
It adds work to every line, so use it to find the slow line, not to time a fix.
Memory numbers that disagree
RSS, or resident memory, is how much RAM the operating system has given
a process. tracemalloc reports traced memory, the memory that Python
allocated while tracemalloc was running. The two can be far apart:
import tracemalloc
before = bytearray(4 * 1024 * 1024)
tracemalloc.start()
after = bytearray(4 * 1024 * 1024)
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(f"held: {(len(before) + len(after)) // 2**20} MiB")
print(f"traced: {current // 2**20} MiB")
This prints held: 8 MiB and traced: 4 MiB. Both buffers are alive, but
tracemalloc never saw the first one.
Check your understanding RSS is 8 GB, but tracemalloc reports 200 MB. Is a C extension leaking 7.8 GB? Show answer Hide answer
Not necessarily. The two numbers count different things. RSS also counts memory that tracemalloc never sees. That includes memory allocated before tracing started and memory that C libraries allocate on their own. It also includes memory Python has freed but kept for reuse. Start tracing earlier, or use Memray to see native allocations, before you decide where the memory went.
Check the fix without the profiler
Every profiler changes how your program runs. Use a profile to find the slow place. Then time your change with the profiler off, on the same workload, as in the earlier timing articles.