Write Tests That Catch Real Bugs

Here is a function, and a test for it. The test passes:

def divide_budget(total: int, people: int) -> tuple[int, int]:
    '''Split total between people. Return each share and what is left over.'''
    if total < 0 or people <= 0:
        raise ValueError("total must be nonnegative and people must be positive")
    return divmod(total, people)


def test_divide_budget() -> None:
    divide_budget(10, 3)

The test also passes if someone breaks the function. Change the last line to return total // people, 0, and the leftover is always 0, but the test still passes. It fails only if divide_budget raises an exception, because it never looks at the result.

Check the result

A test is worth keeping when it fails on a bug someone could plausibly write. Compare the result with the exact answer you expect:

import pytest


def test_budget_keeps_the_remainder() -> None:
    assert divide_budget(10, 3) == (3, 1)


def test_budget_rejects_zero_people() -> None:
    with pytest.raises(ValueError, match="people must be positive"):
        divide_budget(10, 0)

The broken version returns (3, 0), so the first test fails, and pytest prints assert (3, 0) == (3, 1). pytest can show both values because it rewrites the assert statements in your test files as it imports them. The second test checks the error. pytest.raises fails the test if the block raises nothing, or raises a different exception.

Pick inputs where a broken version gives a different answer

divide_budget(9, 3) returns (3, 0), and so does the broken version. A test with those inputs passes for both. For each case, ask which mistake it would catch. The useful cases are usually at the edges:

an exact fit and a near miss, such as 9 and 10 items in groups of 3;
zero and one, such as an empty list or a single item;
the smallest and largest values the function accepts, and the first value it rejects.

The next exercise, Catch the Broken Versions, gives you three broken versions of a function. You pick the cases that catch all three.

Test what callers can see

Callers can see the return value, the exception, and any change to a file or a database. A correct rewrite keeps those the same, so tests that check them keep passing. A test that checks how the code works, such as which helper it calls, fails after a correct rewrite. You then have to update the test for no gain.

Check that the code made a call only when the call is the behavior. "Send the email exactly once" is a promise to the caller, so assert that the fake mailer got one message.

Read coverage as a list of untested lines

Coverage reports which lines ran during your tests. The first test above runs every line of divide_budget except the raise, and checks nothing. So coverage shows where you have not looked yet. It cannot tell you that a test checks anything. Read the lines it reports as missed, and write a test for the ones that matter.

A mutation-testing tool finds tests that run code without checking it. It makes small changes to your code, such as turning < into <=, and runs your tests against each changed version. If the tests still pass on a changed version, an assertion is missing. A later exercise in this track builds a small one.

Fix a flaky test at its cause

A flaky test passes on some runs and fails on others, with no change to the code. The usual causes:

One test changes shared state, such as a module-level dictionary, and a later test depends on it. Give each test fresh state with a fixture.
The test reads the real clock, or sleeps and assumes another thread has finished by then. Pass in a fake clock, or wait for an event.
The test compares an unordered result, such as a set turned into a list, with a fixed order. Compare sets, or sort both sides.

Rerunning a flaky test until it passes hides the bug. Find the cause.

Check your understanding A test asserts that divide_budget(9, 3) == (3, 0). Does it catch the version that returns (total // people, 0)? Show answer

No. 9 divides by 3 exactly, so the broken version also returns (3, 0), and the test passes for both. Use an input with a remainder, such as 10 and 3, where the correct and broken versions give different answers.

Going further

The pytest-cov plugin reports coverage with pytest --cov. Its branch mode, --cov-branch, also reports an if that only ever went one way.

Fakes make tests fast, but they check your code only against your idea of the real service. Keep a few slower tests that use the real database, or the service's test environment, and run them before each release.