Mutation testing: scoring a test suite

“The tests pass” and “the tests are good” are different measurements, and only one of them is routinely taken.

Line coverage measures which lines were executed. It cannot distinguish a test that asserts the right answer from one that calls the function and throws the result away — both light up the same green. Mutation testing asks the question you actually care about: if I broke this code, would the suite notice? Damage the implementation in one small, plausible way, run the suite, and see whether it fails. A mutant the suite fails to catch is a survivor, and it is a precise description of a bug your suite would ship.

You are going to build the scorer.

What to write

def targets(tree: ast.Module, operator: str) -> list[Target]
def mutate(node: Target, operator: str) -> None
def kills(mutant: ast.Module, entrypoint: str, cases: Sequence[Case]) -> bool
def score_suite(*, source, entrypoint, cases, operators) -> Score

Four mutation operators, each applied to one node at a time to produce one mutant:

targets collects the nodes an operator can hit, sorted by (lineno, col_offset, end_lineno, end_col_offset). Determinism matters: ast.walk is breadth-first and its order is an implementation detail, so sorting is what makes one input produce one answer.

kills compiles and executes a mutant, calls entrypoint(**case["kwargs"]) for each case, and reports True if any case raises or returns something other than its expected value. A mutant that raises on import is killed too.

score_suite walks operators in the given order, generates every mutant, and returns:

{"total": int, "killed": int, "score_percent": int, "survivors": list[str]}

score_percent is killed * 100 // total — integer floor division, not round(), which does banker’s rounding and would make 62.5% into 62 while looking like it should be 63. With no mutants at all the score is 100.

Survivors are labelled f"{operator}@{lineno}:{col}" using the position of the node before the mutation, and sorted numerically by (lineno, col, operator) — sorting the strings would put line 10 before line 9.

What the test cases are actually showing you

The same function, grade, scores 28% against a suite that tests 95 and 50, and 100% against a suite that tests 95, 90, 89, 80 and 79. Both suites have identical line coverage: 100%. The difference is that the second one tests the boundaries, which is where the mutants live — and mutation score is the only automatable measurement that can tell those two suites apart.

Then look at the case with an empty cases list. Every mutant survives, so the score is 0% — while a coverage tool run on the same suite would report nothing at all, because nothing ran. That is the shape of the real failure: a test file full of calls and no assertions is invisible to coverage and worthless in production.

And the case where total is 0 is worth a moment too. A vacuous 100% is what every gaming strategy converges on; the fix is that the mutant set is part of the gate, not something the person being measured gets to choose.

A note on the harness

This problem inverts the syllabus’s original framing, where the learner submits a test file and the grader runs it against pre-generated mutants. Single-function submissions cannot express a test file, so you build the grader instead. The lesson survives intact — arguably better, because you end up knowing what a mutation testing tool does rather than only what it says.