Deterministic Concurrency Testing

This item is a cross-reference. The problem lives at 8.10 — Deterministic tests for concurrency, in the testing track, because it is a testing skill before it is a concurrency skill.

The connection is worth stating explicitly, though, because it is the reason this entire track is teachable.

Every problem in this track is deterministic on purpose

Look back at what the exercises actually assert:

9.2 asserts a count — exactly 5,000 — not “roughly 5,000”.
9.3 asserts an ordering property (each consumer’s own subsequence is increasing) and a call count (time.sleep was never called), not a duration.
9.4 asserts a classification (which exception type, which message) and a structural fact (no thread was a daemon), not a timing.
9.9 asserts which siblings observed cancellation, as a sorted list.
9.10 asserts an event log — the exact sequence of body:start, cleanup:start, cleanup:cancelled — not “cleanup ran quickly”.
9.11 asserts an attempt count, which is what makes “retries respect the budget” checkable without measuring wall time.

None of them assert “this took less than N milliseconds”. That is not a limitation of the harness; it is the technique. A concurrency test that asserts a duration is a flaky test, because duration is the one property a shared CI machine will not reproduce.

The three moves

Turn timing into a countable artefact. Instead of “the cleanup was fast”, assert “the cleanup was cancelled” — which is a log entry. Instead of “the retries stopped in time”, assert “exactly one attempt happened”. Instead of “the producer blocked”, assert “the source never got more than queue_size + workers ahead”.

Make the interleaving reachable rather than likely. The 64-threads × 10,000-iterations shape in 9.2 is not there for realism; it is there because a lost update has to be reliably reproducible for the test to mean anything. In async code you get this for free — asyncio.sleep(0) hop counts let you place a task’s suspension points exactly where you need them, which is why every async problem here parameterises hops.

Instrument the seam, not the clock. A shared list appended to under a lock, a counter of predicate evaluations, a set of keys that observed CancelledError — these are all cheap, deterministic and directly meaningful.

💡Your test needs to prove that a worker pool applies backpressure. The obvious test measures how long the producer was blocked. What do you assert instead? click to reveal

Assert how far the source got ahead of the consumers, and bound it structurally.

Instrument the async iterator so it records, at every yield, the number of items produced minus the number completed. Then assert that the maximum of that sequence never exceeded queue_size + workers. That number is not a measurement, it is an invariant of the design: at most queue_size items can be sitting in the queue and at most workers can be in flight, so the source physically cannot be further ahead than their sum.

It is deterministic, it fails loudly on an unbounded implementation (the maximum becomes the total item count), and — unlike a duration — it means the same thing on a laptop and on a loaded CI runner.

The general move: find the invariant the design guarantees, and assert that. “It was fast” is a consequence of the design. The invariant is the design.

Go to 8.10 for the mechanics: deterministic seeds, controllable clocks, asyncio.sleep(0) as a scheduling primitive, and how to make a race reproduce on demand.