Every optimisation claim in a pull request rests on a measurement. If that measurement disabled the garbage collector on code whose cost is garbage collection, or reported a mean polluted by a Slack notification, the claim is noise — and the reviewer has no way to tell, because all they see is a number.
Two documented behaviours of timeit account for most of the bad numbers.
1. timeit turns the garbage collector off
From the documentation: “By default, timeit() temporarily turns off garbage collection during the timing. The advantage of this approach is that it makes independent timings more comparable. The disadvantage is that GC may be an important component of the performance of the function being measured.”
Look at what Timer.timeit actually does:
gcold = gc.isenabled()
gc.disable()
try:
timing = self.inner(it, self.timer)
finally:
if gcold:
gc.enable()For comparing two implementations of a pure function, disabling GC is right: it removes a source of variance neither implementation controls. For measuring anything that allocates heavily — a parser, a serialiser, an ORM materialisation — it is measuring a program you do not deploy. Collection is part of the cost of allocation, and you have removed it.
The documented workaround is to re-enable it inside the setup, which runs inside the timing function, after gc.disable() has already happened:
timer = timeit.Timer(my_function, setup="import gc; gc.enable()")
Now the loop runs with the collector on, and timeit‘s finally still restores the caller’s state afterwards.
2. Take the minimum, not the mean
Also from the documentation, and worth reading twice: “the lowest value gives a lower bound for how fast your machine can run the given code snippet; higher values in the result vector are typically not caused by variability in Python’s speed, but by other processes interfering with your timing accuracy. So the min() of the result is probably the only number you should be interested in.”
The reasoning is that timing noise is additive and one-sided. Nothing makes your code run faster than it can; plenty of things make it run slower — a context switch, an interrupt, another process, a frequency-scaling event. So the sample is a true value plus non-negative noise, and the minimum is the best estimate of the true value.
A mean over seven repeats includes every interruption. A single Slack notification during one repeat moves the mean and does not move the min.
timer = timeit.Timer(candidate)
number, _ = timer.autorange() # 3.6: grow until >= 0.2 s
timings = timer.repeat(repeat=7, number=number) # repeat defaults to 5 since 3.7
best = min(timings)
autorange() picks a loop count so that one repeat takes at least 0.2 seconds, which keeps the clock’s resolution from dominating. In 3.15 the target duration becomes configurable.
💡The docs say take the min. pyperf, the tool the CPython team uses for its own benchmark suite, reports the median with median absolute deviation and compares with a t-test. Both are right. What is the difference in the question being asked? click to reveal
timeit‘s advice answers: which of these two implementations is faster on this machine, right now? You control the machine, you can make it quiet, and you want the value least contaminated by things unrelated to the code. The minimum is the cleanest estimator of “how fast can this go”.
pyperf answers a different question: has performance regressed, across machines, across time, across CPython builds? There the run-to-run variance is not noise to be discarded — it is part of what you are measuring, because it comes from things that also vary in production: ASLR changing cache layout, hash randomisation changing dict collisions, different CPUs, different kernels. pyperf therefore runs many separate processes, uses the median so a single pathological process cannot dominate, reports MAD so you can see the spread, and uses a significance test so it does not declare a 1% move a regression.
The rule that falls out: take the min when comparing two things under identical conditions; take the median across processes when tracking one thing across conditions. Reporting a mean answers neither question well, which is why neither tool uses it.
The typing argument for the callable form
timeit accepts a string or a callable. In a typed codebase they are not equivalent:
timeit.timeit("my_func(1, 2, 'garbage')") # mypy sees a string. Nothing checked.
timeit.timeit(lambda: my_func(1, 2)) # fully checked
A string statement is invisible to every static tool you own. Rename my_func, change its signature, delete it — the benchmark still “compiles” and fails at runtime, or worse, silently benchmarks a different function that happens to still exist. Ruff will not flag the unused import that the benchmark actually needed. mypy --strict gives you exactly nothing.
The callable form is ordinary code. It is checked, refactored, and renamed with everything else.
The one thing the string form does that the callable form cannot is control what happens inside the timing function — which is precisely why the GC workaround above is a setup string. That is the legitimate use, and it is a two-word string that never needs refactoring.
💡You benchmark two dict-building strategies with timeit, get a clean 1.4x win, ship it, and production latency does not move. Give three reasons that is consistent with a correct measurement.
click to reveal
The function was not the bottleneck. A 1.4x win on 2% of request time is 0.6%. Microbenchmarks measure what you point them at; profiles tell you where to point them. This is why cProfile comes before timeit in any sane workflow.
The GC was off. If the faster version allocates more — a comprehension instead of a generator, more intermediate dicts — timeit hid the cost of collecting it, and production did not.
The inputs were unrepresentative. A benchmark over ten keys says nothing about a hundred thousand, and a benchmark over uniformly random keys says nothing about the clustered, repeated, low-cardinality keys real traffic produces. Hash collisions, branch prediction and cache behaviour all depend on the data.
There is a fourth worth naming: the specialising interpreter needs a few iterations to warm up, so a tight timeit loop measures fully specialised code that a production call path — invoked once per request, from varying call sites — may never reach.