The profiling toolbox beyond the stdlib

The standard library covers three of the four questions you will be asked in an incident:

Which function is slow? cProfile / profiling.tracing.
Where is this running process spending time, right now? profiling.sampling — 3.15 only.
Which Python line allocated this memory? tracemalloc.

It does not cover: which line inside a known-slow function; which native allocation is holding 6 GB; or the second question at all if you are on 3.12 through 3.14, which most production Python is.

Four tools fill those gaps. Knowing which one answers which question is most of the value; the rest is reading their README.

py-spy — sampling, attach, works today

A sampling profiler that attaches to a running process by PID with no instrumentation and no code change. This is the answer for a 3.12-3.14 target, which is to say: the answer, now.

py-spy dump --pid 12345        # instant stack snapshot of every thread
py-spy top --pid 12345         # live, top-like view
py-spy record --pid 12345 -o flame.svg

py-spy dump is the single highest-value command in this article. A process that appears hung, one command, and you have every thread’s stack — no debugger, no restart, no reproduction. Learn it before you need it.

memray — the one that sees native allocations

Bloomberg’s memory profiler, and the tool that answers the question tracemalloc cannot. It tracks allocations in Python code, in native extension modules, and in the interpreter itself, with the full native call stack.

That is the difference between “your NumPy buffers are invisible” and “here is the C++ frame that allocated 6 GB”. A service whose RSS is 8 GB and whose tracemalloc total is 200 MB is telling you the memory is not in Python objects; memray is how you find out where it actually is.

It has a live mode, can attach to a running process, and ships a pytest plugin so you can assert memory bounds in tests.

Linux and macOS only. No Windows.

scalene — CPU and memory in one pass, Python separated from native

Profiles CPU and memory together and — the distinguishing feature — attributes time to Python versus native code separately. That is directly the question behind “should we rewrite this in C?”: if 90% of the time is already in native code, the answer is no, and scalene tells you that in one run rather than after a week of Cython.

Line-level, and it will also point at GPU time where applicable.

line_profiler — inside one known-hot function

@profile
def parse_batch(rows: list[bytes]) -> list[Record]:
    ...

Per-line timings with hit counts. The overhead is high enough that you do not run it broadly; you run it on the one function cProfile already identified, when the function is long enough that “which line” is a real question.

The workflow

Sampling profiler in production (py-spy record, or profiling.sampling on 3.15) — find where the time actually goes, on real traffic, under real load. Not on your laptop, where the input distribution is wrong.
Reproduce locally, guided by what you saw. This step is where most of the work is, and skipping it means you cannot verify a fix.
line_profiler or scalene on the specific function, when “which function” is not a fine enough answer.
timeit on the candidate change, unprofiled, to find out whether it actually helped (article 11.17).
Sampling profiler again, in production, to confirm the shape moved. Local improvements do not always survive contact with real data.

Step 5 is the one people skip, and it is the one that catches the change that was 1.4x faster on a synthetic input and neutral on real traffic.

💡Your service's RSS is 8 GB. tracemalloc accounts for 200 MB. Walk through what you would do next, and what each step would rule out. click to reveal

The gap itself is the first finding: 7.8 GB is not in Python objects. That immediately rules out the whole category tracemalloc is good at — no dict is too big, no cache is unbounded, no list is being appended to forever. Whatever this is, it did not go through PyMem_Malloc.

So: memray, in attach mode if the process is live, or under memray run on a reproduction. It sees native allocations with native stacks, so the output will name a library rather than one of your functions. That is the point — the answer here is nearly always “a C extension”, and the value is knowing which.

The three usual answers, and what each implies: a NumPy or PyTorch buffer that something is retaining, which is a Python-side reference problem with a native-side symptom; an allocator arena that has been freed but not returned to the OS, which is fragmentation and not a leak at all; or a genuine leak in a C library — a driver, an image codec, a compression library — which you cannot fix and must work around by bounding the process’s lifetime or upgrading the dependency.

And one thing to check before any of that, because it is free: whether the process is pre-forked. If eight workers each report 1 GB of RSS and they were forked from a parent that loaded a 900 MB model, the “8 GB” may be the same pages counted eight times by a tool that does not understand sharing. Check PSS rather than RSS before profiling anything (article 11.5).