Someone opens a pull request titled “3% faster JSON path”. The change adds a lookup table, three branches and a comment explaining the lookup table. The benchmark in the description shows 3%. Should it merge?
On the evidence given, there is no way to know, because a single wall-clock measurement on a developer laptop cannot resolve 3%. pyperf’s own documentation reports a maximum 39% above the mean on an untuned system. Against that background, 3% is not a small effect — it is not an effect at all.
And the cost of merging it is permanent: three branches and a lookup table that every future reader has to understand, forever, in exchange for a number that may be zero.
What moves your number without touching your code
CPU frequency scaling and turbo. A modern core boosts above its base clock while thermal and power headroom allow, then drops. A benchmark that starts on a cool core and ends on a hot one measures the cooling system. The first variant you measure often wins for this reason alone, which is why interleaving A and B rather than running all of A then all of B matters.
Other processes and interrupt handling. Every context switch evicts your working set from L1 and L2. An IRQ handler on your core runs before you do. On a laptop that is a browser tab; on a shared CI runner it is another job.
NUMA locality. On a multi-socket machine, memory allocated on one node and read from a core on another costs meaningfully more. Whether your process lands that way is scheduler luck.
ASLR. Address space layout randomisation changes where your code and data sit, which changes cache-set conflicts and branch-predictor aliasing. The same binary on the same input has a run-to-run distribution from this alone, and it does not shrink with more iterations inside one process — it is a per-process effect.
Hash randomisation. PYTHONHASHSEED is random per process by default, so string hashes differ, so dict and set iteration order and collision patterns differ. For code that iterates dicts or hits hash collisions, this is a real per-process variance source.
Notice that three of those — ASLR, hash randomisation, NUMA placement — vary between processes and are constant within one. So running more iterations inside a single process gives you a tighter distribution around a possibly-wrong centre. That is the deep reason pyperf spawns many processes and timeit alone is not enough for cross-machine work.
pyperf system tune
pyperf ships a command that addresses these directly: it pins to isolated CPUs, disables turbo boost, sets the performance governor, disables address space randomisation and fixes the hash seed. Its documentation notes that CPU isolation in particular has “a significant impact on the stability of benchmarks”.
You will not do this on your laptop. You should do it on the machine that gates merges, if you have one.
💡timeit says take the minimum. pyperf uses the median with median absolute deviation, across many processes, with a t-test. Present the disagreement honestly: when is each right?
click to reveal
They are answering different questions, and both answers are correct for their question.
timeit‘s minimum is right for A-versus-B on one quiet machine, now. Noise is additive and one-sided, so the minimum is the least-contaminated estimate of “how fast can this code go here”. You are comparing two things under identical conditions and you want the conditions removed from the answer.
pyperf’s median-across-processes is right for tracking one thing over time and across machines. Here the between-process variance — ASLR, hash seed, NUMA placement — is not contamination to discard; it is part of the distribution your users experience, and a change that is only fast under a lucky memory layout is not a change. Taking the minimum across processes would systematically report the luckiest layout, which is not a number that predicts anything. The MAD tells you the spread, and the t-test is what stops a 1% wobble being announced as a regression.
The failure mode of confusing them is asymmetric and worth naming. Using the median for an A/B on your laptop just makes you slightly less sensitive. Using the minimum for a regression-tracking suite makes it report improvements that are pure luck and miss regressions that are real.
Warm-up matters more in 2026 than it used to
Two mechanisms in current CPython make the first iterations unrepresentative:
The specialising adaptive interpreter (PEP 659, 3.11). Bytecodes rewrite themselves into specialised forms once they have seen a stable type — LOAD_ATTR becomes a cached slot read, BINARY_OP becomes an int-specific add. That takes iterations. A benchmark that runs a function ten times measures unspecialised code; production, which runs it ten million times, does not.
The tracing JIT (3.13+, off by default). It compiles nothing until a loop is hot. If it is enabled at all (PYTHON_JIT=1), a short benchmark measures the interpreter and reports it as the JIT’s performance.
Both push the same way: discard the first iterations, and make sure the loop count is large enough that the steady state dominates. autorange() does some of this for you by choosing a count that takes at least 0.2 seconds.
The opposite error is worth naming too. If the code you care about genuinely runs once per request from a cold-ish call site, then the warmed-up steady-state number is the misleading one. Benchmark the state you deploy.
💡What would you require in a pull request description before accepting a performance claim? click to reveal
Four things, and none of them is a single number.
The measurement method, precisely: which tool, how many repeats, how many processes, what statistic, and whether the GC was on. “1.4x faster” with no method attached is not reviewable.
The variance, not just the centre. A median with a MAD, or a min with the spread of the samples. If the improvement is smaller than the run-to-run spread, there is nothing to discuss.
The environment: the machine, whether it was tuned or idle, the Python version and build (free-threaded? JIT enabled?), and the input data. Inputs matter more than people expect — a dict benchmark on random keys says nothing about the clustered low-cardinality keys real traffic produces.
Where it sits in a profile. A 1.4x win on 2% of request time is 0.6%, and the honest response to that is usually to close the PR. This is the question that most often ends the discussion, and it is the cheapest one to ask.
The meta-point: the reviewer’s job is not to verify the number. It is to establish that the number could in principle have been wrong and was not. A measurement that could not have failed to show an improvement has not demonstrated one.