A benchmark answers a specific question about a particular workload and build. Before comparing numbers, decide what work you want to measure: a function call, a complete request, process startup, or steady-state throughput. Those measurements can all be useful, but they include different costs.
What this site's Rust timing includes
The current Rust daemon compiles the submission first. For benchmarking, it discards one full pass over the test cases, then takes the median of three timed passes. Each case runs in a separate process. A timed pass includes process startup, the case's input/output handling and validation, as well as the submitted code. Compilation is outside that interval.
This is a whole-harness measurement, not the duration of just your function. For short exercises, process overhead can dominate. The reported process overhead is diagnostic information; it is not subtracted from the median.
Where a stored reference runtime is available, the site compares your time
with that reference. Hardware, toolchain and machine load can affect the
comparison. Equal additive overhead would not cancel in a ratio: if useful
work takes 1 ms versus 2 ms, adding 10 ms to each changes the ratio from
1 / 2 to 11 / 12. These are illustrative numbers, not measurements.
Use the site's result as feedback under those conditions. For a precise comparison of two implementations, benchmark both under the same controlled setup and inspect repeated measurements.
Keep the intended work observable
An optimizer may remove a pure calculation whose result is unused, or simplify it when the inputs are known. Timing a loop does not prove that every source-level operation survived compilation.
std::hint::black_box
helps prevent optimizations based on the benchmark's surrounding context.
Use it on inputs before the operation and on results afterward. Here is a
complete, minimal example:
use std::hint::black_box;
use std::time::Instant;
fn sum(values: &[u64]) -> u64 {
values.iter().copied().sum()
}
fn main() {
let values: Vec<u64> = (0..1_000).collect();
assert_eq!(sum(&values), 499_500);
// Warm this example's workload before timing repeated calls.
black_box(sum(black_box(values.as_slice())));
let start = Instant::now();
for _ in 0..10_000 {
black_box(sum(black_box(values.as_slice())));
}
println!("10,000 calls: {:?}", start.elapsed());
}
This measures a batch of calls over the same input. Input construction and printing are outside the interval; loop and black-box overhead remain inside it. Reusing the input also makes this a warm-data workload. It is not a model of every application that sums arrays.
black_box preserves the value and offers best-effort optimization
resistance. Its effectiveness depends on the platform and compiler backend.
It is not a guarantee that a particular instruction will execute, and it
provides no constant-time cryptography guarantee.
In black_box(5 * 10), multiplication can still be folded before its result
reaches black_box. Making an operand opaque first changes that situation.
Literal-derived test data is not inherently invalid; what matters is whether
the compiler has knowledge that your real workload would not provide.
Choose warm or cold measurements deliberately
First runs may include page faults, initialization, cache effects or other startup costs. Warm-up can help when the question is steady-state behavior. If startup latency is the question, keep and measure those costs instead. There is no universal first-run delay or rule to discard every first sample.
Repeatedly processing one input can improve cache reuse and branch prediction, but does not guarantee that the data fits in cache or every branch is predicted. Include input sizes, shapes and reuse patterns that match the use you care about.
Look at variation, not just one number
Repeat before-and-after measurements. Alternate or randomize their order where practical so temperature, background work and system drift are less likely to favor one version. Record the build flags, Rust version, target and relevant hardware alongside the results.
A median is less sensitive to a few unusually slow samples than a mean, but it does not remove noise or make a small sample conclusive. Inspect the spread. For a service, slow-tail latency may be part of the result you need rather than something to discard. Avoid claiming a speedup when the observed difference is difficult to distinguish from normal variation.
Keep the build configuration comparable
Optimization level, target CPU, link-time optimization and codegen-unit
settings can change generated code. An unrelated-looking edit can affect
layout or inlining too. Do not attribute every timing change to the
algorithm, or assume an #[inline] annotation guarantees an improvement.
Use a production-like optimized build when estimating production performance. Debug-build measurements can still answer questions about debug builds; they simply do not establish release performance. Inspect profiles or generated code when a result needs explanation.
Count allocations when that answers the question
An allocation counter can reveal repeated buffer growth or temporary strings. It measures calls observed by that instrumentation in that build, not a universal property of the source code.
Define what you count: allocation requests, reallocations, requested bytes, peak live bytes, or something else. Keep logging and unrelated threads out of the measurement, and account for setup and cleanup consistently.
Do not promise fixed counts for “1,000 pushes” without specifying the
element type, starting capacity, implementation and counting method. Growth
strategies and optimizations can change the result. The
GlobalAlloc documentation
explicitly warns that an optimizer may eliminate allocations or move them
to the stack, even when a custom allocator tracks calls.
Reserving adequate capacity can avoid growth while filling a vector. Reusing a string buffer can avoid repeated temporary strings. Verify the effect in the workload you are testing. Fewer allocations do not by themselves prove lower latency, and a fixed exercise budget is a condition of that grader, not a promise across all Rust builds.
Check your understanding A JSON parser benchmark reports 2 GB/s. What would make that number useful? Show answer Hide answer
Ask what documents were parsed, whether the same input was reused, what output was produced and consumed, and which work was timed. Include or exclude allocation, validation, buffer reuse and output traversal according to the comparison you intend to make.
Then check the build, machine, sample count and variation. A reused arena is representative if the application also reuses one; it is misleading if callers allocate a fresh result every time. A warm small document and a varied stream of large documents are different workloads, not competing answers to one universal parser-speed question.
Confirm effects in the application
A microbenchmark isolates a small piece of work. In an application, that work shares caches, memory bandwidth and CPU time with other code, and may account for only a small fraction of total latency. Measure end-to-end behavior before claiming an application-level improvement.
Microbenchmarks remain useful for comparing implementations and investigating specific costs. They complement application measurements rather than replacing them.
Tools available here and in a Cargo project
This site's Rust exercises use a single-file, standard-library-only harness.
You can use Instant, black_box and carefully designed counters within
that environment, while preserving the exercise's required output format.
Do not paste the example's timing output into a graded function.
In a separate Cargo project, cargo bench
works on stable Rust with a custom benchmark harness. The built-in
#[bench] attribute is the unstable part. Benchmarking libraries such as
Criterion can provide a custom harness and statistical tools, but choosing
one does not make the workload or interpretation correct automatically.