This platform times your submission and shows you a ratio. That number is genuinely useful and it is genuinely easy to fool. Before the rest of this track hands you levers, you need to know what the lever is attached to.
Three of the measurements quoted in this course came out the opposite of folklore, and one showed no difference at all where a difference obviously should have existed. Every one of those was a methodology story, not a Rust story.
1. Dead-code elimination: the benchmark that measured nothing
let start = Instant::now();
for _ in 0..1_000_000 {
expensive(input);
}
println!("{:?}", start.elapsed());
If expensive is pure and its result is unused, LLVM may delete the call, then the loop, then the whole block. You have measured Instant::now() twice. The symptom is a suspiciously round, suspiciously tiny number — nanoseconds for work that cannot possibly be that fast.
The tool is std::hint::black_box:
let mut acc = 0;
for _ in 0..1_000_000 {
acc ^= black_box(expensive(black_box(input)));
}
black_box(acc);It is an opaque identity function: the value goes in, the same value comes out, and the compiler is asked to assume it cannot see through it.
💡black_box is documented as "best-effort". What exactly is not guaranteed, and what is the concrete consequence for a benchmark you write today?
click to reveal
Three things are not guaranteed, and each has bitten people.
It is not a barrier the optimiser is forbidden to cross. The documentation says the function is best-effort and its effectiveness is platform-dependent — it is implemented as an empty inline-assembly block with the value marked as an input, and how much that constrains LLVM varies by backend and by version. It is not part of Rust’s stability guarantees, so a compiler upgrade can change how much it hides.
It is explicitly not for constant-time cryptography. The docs say so in as many words. Learners treat black_box as “the compiler cannot see this”, conclude it prevents timing side channels, and ship a black_box-based comparison as constant-time. It is not. Use a crate designed for the job.
Its input is still optimised. This is the one that catches people writing benchmarks. black_box(5 * 10) still folds to black_box(50) — the multiplication happens at compile time, and only the result is hidden afterwards. To stop the folding you must make the input opaque before the operation, which is why the loop above wraps input as well as the result.
Practical rule: hide the inputs on the way in and the outputs on the way out, and treat a suspiciously fast result as a bug in your benchmark until proven otherwise.
2. Constant folding: literals are not data
let v: Vec<u64> = (0..1000).collect();
let s: u64 = v.iter().sum();
Both lines are visible to the optimiser from end to end. LLVM can, and sometimes does, compute the answer at compile time and emit a single mov. You then report that summing a thousand integers takes 0.3 ns.
Anything derived from a literal is suspect. Read the input from stdin, from a file, from a runtime argument — or at minimum push it through black_box before the code under test touches it. Every benchmarked problem in this track takes its size and seed from the test input for exactly this reason.
3. Warm-up, and what the first run really measures
The first execution of a freshly built binary on macOS costs about 250 ms — code-signature validation — against about 2 ms thereafter. Beyond that there is page-faulting the binary in, filling the branch predictor’s history, populating the caches, and letting the CPU’s frequency governor settle.
Discard the first iteration. Always. This harness does: its benchmark runs a warm-up pass and then takes the median of several timed passes.
Median, not mean, because the distribution is not symmetric. A benchmark’s fast times cluster tightly and its slow times have a long tail — a scheduler preemption, an interrupt, a background process. The mean chases the tail; the median ignores it.
4. A single number is not a result
Run the same benchmark five times and you will get five numbers. If your “1.4× improvement” is smaller than the spread between runs of the unchanged code, you have measured noise and written a changelog entry about it.
The minimum honest practice: run before and after several times each, look at the spread, and only believe a difference that is clearly larger than it.
5. The codegen-unit wobble, specific to this harness
rustc -O defaults to 16 codegen units, and which functions land in which unit is not something you control. Moving a function, renaming it, or adding an unrelated one can change the partitioning — and therefore change which calls get inlined across unit boundaries.
Measured in this course: a small hash function in a hot loop ran 0.604 ms with no attribute and 0.401 ms with #[inline], a 1.5× difference inside a single file. With -C codegen-units=1 the same two came out 0.367 and 0.372 — no difference at all.
So an edit that “should not have mattered” genuinely can, here, and the cause is the build configuration rather than your algorithm. This is a real source of edit-to-edit noise in this harness specifically, and it is worth knowing before you spend an hour explaining a 5% swing.
6. The deterministic alternative: count, do not time
Timing is the wrong instrument for most of what you actually want to know. Allocation counts are exact, reproducible, machine-independent and explainable:
| allocations | |
|---|---|
Vec::new() + 1000 pushes |
9 |
Vec::with_capacity(1000) + 1000 pushes |
1 |
format!-accumulating 200 parts |
399 |
with_capacity + push_str, same output |
1 |
Every one of those numbers is derivable from first principles, identical on every machine, and identical on every run. A custom GlobalAlloc that increments a counter is twenty lines, and it is this platform’s own methodology — several problems in this track are graded on allocation budgets rather than on time, precisely because a time-based gate would be a coin flip.
Use timing to find out whether something is slow. Use counting to find out why, and to write the regression test.
💡A colleague says: "I benchmarked our JSON parser and it does 2 GB/s. Ship it." What questions would you ask before believing that number? click to reveal
Roughly in order of how often each one turns out to be the problem.
What input? If it is one document parsed a million times, it lives in L1 cache after the first iteration and every branch is perfectly predicted. Real documents arrive cold, from a socket, varying in shape.
Was the result used? If the parsed value is dropped without inspection, the parser may have been partially eliminated — and even if not, dropping is often cheaper than the traversal a real caller performs.
Was the input a literal? A const JSON string is visible to the optimiser end to end.
Median or mean, and over how many runs? What was the spread? “2 GB/s” from one run means nothing.
Does it include allocation and setup? Parsers that build owned trees are dominated by allocation; a benchmark that reuses one arena reports a number no production caller will ever see.
Is the machine representative? Same architecture, same core count, same thermal state, no debug build (a debug-build benchmark tells you nothing about release).
And then the deepest question, which is the subject of the next section.
7. The problem you cannot fix by being careful
A microbenchmark measures a function in isolation — and isolation is exactly the condition under which the surrounding program’s costs disappear.
In your benchmark, that function has the entire L1 cache, the whole branch-predictor table, and every register to itself. Inside a real program it competes with everything else: its data has been evicted, its branches have been displaced by other branches’ history, its instruction footprint pushes something else out.
Which is why an optimisation can be 3× in a microbenchmark and 0% in production, and — less often but more surprisingly — why a change can be neutral in a microbenchmark and a real win in production because it made the code smaller.
Nothing about your methodology fixes this. The only answer is to also measure the whole system: end-to-end latency, throughput under real load, the number your users actually experience. Microbenchmarks are for comparing two implementations of the same thing. They are not for predicting the effect on your program.
8. What you cannot use here
#[bench] and cargo bench are nightly-only, and Criterion — which everybody uses in real projects, and which does the statistics properly — is an external crate. Neither exists in a single-file, no-cargo harness.
So the honest tools available here are exactly three: std::time::Instant, std::hint::black_box, and the counting allocator. That is enough to be rigorous, and this article is what makes it enough.
The checklist
-
Hide inputs and outputs with
black_box, and remember its input is still folded. - Do not derive the input from a literal.
- Discard the warm-up run.
- Take a median over several runs, and look at the spread before believing a difference.
- Never benchmark a debug build.
- Prefer a deterministic counter to a stopwatch whenever the question allows it.
- Confirm the microbenchmark’s verdict end to end before you believe it about your program.
Related lints, all of which fire on benchmark-shaped mistakes: no_effect (a statement that does nothing — often the tell that the compiler deleted your work), unit_arg, and in pedantic let_underscore_untyped and unused_self.