Skip to content

← Performance step 6 of 7

Medium Primitives

Measure runtime after warmup

Jump to the task

Decide what you want to measure before starting the clock. The first call can include initialization or compilation, so its latency may differ greatly from later calls. That startup cost matters for some applications. For steady-state performance, warm up first and measure repeated calls.

Why timings vary

Depending on the workload, early calls can initialize device resources, allocate buffers, choose algorithms or compile code. These costs are not necessarily one-time: a new shape or compilation guard can trigger more work. Warm up with representative inputs and check that the timings have settled.

Repeated measurements reveal variation that one number hides. The median is less sensitive than the mean to a few extreme samples, but it does not remove systematic bias, timer overhead or thermal effects. Keep the conditions and sample count alongside a result you report.

Wait for asynchronous work when timing completion

A CPU clock around an asynchronous CUDA launch may stop before the device finishes. For an isolated wall-clock measurement, synchronize the relevant CUDA device before starting the timer and after the work, before stopping it. That measurement includes host overhead and the wait for completion.

CUDA events offer another way to measure elapsed time on a device stream. Choose a method that matches the question: synchronizing every batch changes the overlap in a throughput workload. Other accelerators have their own synchronization APIs; a CPU-only callable does not need CUDA synchronization.

A benchmarking utility

For repeated measurements, torch.utils.benchmark.Timer handles warmup, thread settings and supported accelerator synchronization. Its blocked_autorange method collects repeated timing blocks while limiting timer overhead. For example:

import torch
from torch.utils.benchmark import Timer

x = torch.ones(1024)
measurement = Timer(stmt="x + 1", globals={"x": x}).blocked_autorange(
    min_run_time=0.05
)
print(measurement.median >= 0)  # True; the actual time depends on the machine.

Your task

def measure(fn, warmup: int, runs: int) -> dict

Implement a small CPU timing loop. warmup is nonnegative and runs is positive. Call fn warmup times without reading the clock. Then, for each of runs calls, read time.perf_counter immediately before and after fn and record the difference. Return the median duration in seconds under median, and warmup + runs under calls.

The harness replaces perf_counter with a controlled clock and gives each fn call a known duration. It checks the call order, the reported count and the median, so passing does not depend on how busy the machine is. This exercise checks the timing algorithm, not GPU synchronization or benchmark quality.