Test selection: filters, #[ignore], and the libtest CLI

Everybody learns cargo test and then flails the first time a suite gets slow or one test needs debugging in isolation. The rules are simple, precisely specified and constantly needed.

pub fn select(
    tests: Vec<(String, bool)>,
    filters: Vec<String>,
    skip: Vec<String>,
    exact: bool,
    ignored_mode: String,
) -> Vec<String>

Each test is (full_path, is_ignored). Return the names that would run, in input order.

libtest’s algorithm

1. Include a test if it matches any filter — or if there are no filters at all.
2. Exclude it if it matches any skip entry, under the same matching rule.
3. Then apply the ignore mode:
   • "default" keeps only non-ignored tests,
   • "ignored" keeps only ignored tests,
   • "include_ignored" keeps both.

“Matches” means substring of the full path — or full equality when exact is set. Anything other than "ignored" or "include_ignored" behaves as "default".

The over-matching trap

cargo test adds does not run the test called adds. It runs every test whose full path contains adds, which on a real project is math::adds, math::adds_negatives, bench::adds and probably three more you forgot about. That is usually helpful and occasionally maddening. cargo test -- --exact tests::math::adds narrows it to one.

Note where that -- went, because this is the thing that trips everybody once:

Arguments before -- go to Cargo. Arguments after -- go to the test binary.

cargo test --nocapture is an error — Cargo has no such flag. cargo test -- --nocapture works. Same for --exact, --test-threads, --ignored, --include-ignored, --list.

While we are on flags that look alike: -j sets how many compile jobs Cargo runs; --test-threads sets how many tests run concurrently. Only the second one has anything to do with flaky ordering, and --test-threads=1 is the first thing to try when a suite passes alone and fails together.

#[ignore] and why it is not #[cfg]

An ignored test is compiled, listed and skipped. That matters: it still has to type-check, so it cannot rot the way a #[cfg(feature = "slow")] block can. Run them with cargo test -- --ignored, or run everything with --include-ignored.

The reason this site returns artefacts instead of measuring

Tests run in parallel by default, in one process, sharing everything process-global — environment variables, the current directory, the panic hook. Two tests that both call env::set_var are a race, not a suite. This is the single most common source of “passes on my machine”.

It is also the reason every problem here is a deterministic pure function over JSON: a grading harness that measured durations or observed thread interleavings would be grading the machine, not the submission. When you need to test concurrency, return an artefact — a final count, a sorted merge, an event log — never a duration.