Every other item in this track is a technique. This one tells you which technique to apply, and it is the one people skip. The result of skipping it is entirely predictable: you optimise the 2%, spend a day, and report a 0.3% win.
The discipline, which matters more than the tools
- Profile first. Not “read the code and guess.” Not “I know what’s slow.”
- Form a hypothesis. “This function is 40% of the time because it allocates per element.”
- Change one thing.
- Re-measure. Same input, same machine, same methodology as the previous item.
- Repeat, or revert.
Steps 3 and 5 are where discipline breaks. Two changes at once means you cannot attribute the result, and a change that did not help must actually be reverted rather than left in “because it can’t hurt”. It can: it costs readability forever and it makes the next profile harder to read.
The single most common beginner mistake
Profiling a debug build tells you nothing about a release build.
Not “less”, not “roughly the same shape”. Nothing. Debug builds keep every bounds check, keep every Option wrapper as a real branch, inline nothing, and leave every generic call as an actual call. A function that dominates the debug profile can vanish entirely from the release binary because it got inlined and folded into its caller.
Profile --release. Every time.
Which then creates the second problem
Release builds strip debug info by default, so your flamegraph is a wall of hex addresses and ??. The fix is two lines:
[profile.release]
debug = true # or `debug = 1` for line tables onlyand, when stacks still look wrong,
RUSTFLAGS="-C force-frame-pointers=yes"
debug = 1 does not slow the code down. It emits debug information alongside the same optimised instructions; the binary is larger on disk and identical in behaviour. There is no reason not to have it on for any build you might ever want to profile.
💡Your release flamegraph shows 60% of the time inside a function you are certain should be trivial — a two-line accessor. What are the two most likely explanations, and how do you tell them apart? click to reveal
Explanation one: inlining attribution. Sampling profilers record the instruction pointer. When function b is inlined into a, b‘s instructions are a‘s instructions, so the samples get attributed to whichever frame the unwinder reports. Depending on the tool and the debug info available, a big inlined callee’s cost can land on the small caller’s name — or a small caller can appear to own work that came from a callee three levels down.
Explanation two: it really is hot, because of what it touches. A “trivial” accessor that dereferences a pointer into a cold data structure spends its time on a cache miss, not on its two lines. The instruction is cheap and the memory access behind it is 200 cycles.
How to tell them apart:
-
Ask for inlined frames.
perf report --inline, or a profiler that reads DWARF inline records, will expand the inlined callees under the frame. If the time redistributes, it was attribution. -
Use
#[inline(never)]as a diagnostic. Put it on the suspect function, rebuild, re-profile. If the frame now shows a small self-time and a large child, you had attribution. This is a legitimate technique and the reason#[inline(never)]exists as more than a curiosity — take it off when you are done. -
Check cache counters.
perf stat -e cache-misses,cache-referencesorcachegrind. A cache-miss-dominated accessor is a data-layout problem, and the flat-array item in this track is the fix.
The general lesson: a sampling profiler tells you where the instruction pointer was, which is not the same question as “which of my functions is responsible”.
The tools
CPU sampling
-
perf(Linux) — the baseline.perf record -g --call-graph dwarf ./target/release/appthenperf report. Everything else is a nicer front end. -
samply— cross-platform (macOS and Linux), sampling, opens the results in the Firefox Profiler UI. The lowest-friction good answer in 2026, and the one to reach for first. -
cargo flamegraph— wrapsperf/dtraceand produces the flamegraph SVG directly. Width is time; the interesting shapes are the wide plateaus, not the tall spikes. - Instruments (macOS) — the Time Profiler template works on Rust binaries and gives you the best UI of the lot, plus system-level context.
Simulation and counters
-
valgrind --tool=callgrind— counts instructions rather than sampling time. Slow (50–100×) and deterministic, which makes it uniquely good for detecting a 2% regression that sampling noise would bury.kcachegrindreads the output. -
valgrind --tool=cachegrind— cache hit and miss rates per function. This is how you confirm a data-layout hypothesis rather than assuming one.
Allocation
-
dhat(a valgrind tool, and also a Rust crate) — where the allocations are, how big, how long-lived. The counting allocator introduced in the next item is the same idea in twenty lines and no dependencies.
Size and code bloat
-
cargo llvm-lines— how many lines of LLVM IR each generic instantiation generates. The tool for finding monomorphisation costs. -
cargo bloat— which functions and crates own your binary size.
None of these exist in this harness. It compiles one file with rustc and no cargo, so cargo-prefixed tools are out by construction and the valgrind/perf family are out because nothing here runs your binary under a supervisor. That is the honest reason this item is an article: the workflow is real and it lives outside this box.
Allocation: chase it only when the profile says so
The Rust Performance Book quotes a useful rule of thumb: reducing allocation rates by about 10 allocations per million instructions yields roughly a 1% speedup.
Read that in both directions. It means allocation is worth chasing when your program allocates heavily — and it means that removing three allocations from a function that runs once is not an optimisation, it is a diff.
The measurements elsewhere in this track show both sides. A per-record .clone() in a scan over 100 000 records cost 6.3×; that is thousands of allocations per million instructions and it dominates. A Vec<Vec<f32>> versus a flat Vec<f32> over a million elements cost only 1.13× in time, because 1001 allocations amortised over a bandwidth-bound loop is noise — even though the structural difference is dramatic.
💡You profile and find the time is spread evenly across two hundred functions, none above 2%. What does that tell you, and what do you do? click to reveal
It usually means one of four things, and they call for different responses.
The cost is in something that is not a function. Cache misses, branch mispredictions and TLB pressure spread across everything that touches the data. perf stat will show it: a low instructions-per-cycle number with a high cache-miss rate is a data-layout problem, not a hot-function problem. The fix is structural — flatten the data, shrink the types, improve locality — and it moves every one of those two hundred functions a little.
Everything got inlined. In a heavily generic Rust program the profile can be a fog because the call graph collapsed. Build with -C inline-threshold lowered, or add #[inline(never)] to a few suspects, purely to get a readable profile.
The algorithm is the problem. A flat profile with no hot spot often means you are doing O(n²) work spread over many small helpers. No amount of micro-optimisation fixes that; step back and count operations rather than measuring them.
It is genuinely fast enough. A flat profile on a program that meets its latency budget is a finished program. The correct action is to stop, and this is more often the right answer than people like.
What you should not do is start optimising the 2% function at the top. Two percent is your ceiling, and you will not reach it.
The one-line version
Measure, do not guess; measure the release build, or you have measured nothing; change one thing; and know that a sampling profiler answers “where was the instruction pointer”, which is a subtly different question from the one you asked.