Every team that takes testing seriously eventually writes too many of the wrong tests, and the symptom is specific: refactoring becomes expensive. You change how something works without changing what it does, and forty tests fail. The suite is no longer protecting you from regressions; it is protecting the current implementation from you.
Four habits produce that outcome. All four look like diligence.
Testing private functions
def test_normalise_key() -> None:
assert _normalise_key(" Foo Bar ") == "foo_bar"
_normalise_key is private. That underscore is a promise you made to yourself: this can change without notice. The test is now a caller, and it has cancelled the promise.
Six months later you inline _normalise_key, or split it in two, or replace it with a lookup table. The public behaviour is identical. The test suite goes red, and someone has to decide whether to fix the tests or revert the refactor — a decision that should never have existed.
There is a real objection: the private function is where the tricky logic is, and testing it through the public API takes six lines of setup. That objection is usually correct and it is usually pointing at something else. A private function with enough logic to want its own tests is a module-scale idea trapped inside a class, and the answer is not to test it privately but to promote it: give it a name, put it in its own module, make it public, and give it the API a small public thing deserves. Then the tests are legitimate, because there is now a promise to keep.
The heuristic that holds: test what a caller could depend on. If nobody outside could observe it, testing it converts an implementation detail into an interface.
💡Where does that leave the test for a private algorithm that genuinely has no public seam — a scoring heuristic buried three layers deep in a recommender? click to reveal
Two honest options, and one that feels like a compromise and is not.
Option one: promote it. A scoring heuristic is a pure function from features to a number. That is a module. scoring.py with a public score(features: Features) -> float can be tested directly, property-tested, benchmarked, and swapped — and the recommender that calls it gets simpler. Most “no public seam” situations dissolve on inspection, because the thing you want to test is the one part that had no business being private.
Option two: accept the setup cost and test through the boundary. Write the six lines. Use a factory fixture so the sixth test costs one line rather than six. What you get in exchange is a test that survives every refactor of the internals, which is the entire point. If the setup is truly heavy, that heaviness is data — it is telling you how tangled the dependency is, and it will be the same heaviness a future maintainer faces.
The non-option is to test the private function and add a comment saying so. It reads as a considered decision and behaves exactly like an unconsidered one: the coupling is identical, and the comment does not fail the build when someone refactors.
There is a third case worth naming: sometimes you want to test a private function temporarily, while building something, and delete the test afterwards. That is fine and nobody does it, because tests are additive by culture. If you write one, put the deletion in the same PR description as the thing it was scaffolding for.
Coverage as a gate
Coverage measures which lines executed. It cannot distinguish this:
def test_pipeline() -> None:
result = run_pipeline(fixture_input)
assert result is not Nonefrom a test that checks the output. Both light up the same lines, and the first one is worth approximately nothing — it catches only crashes, which is the one failure mode you would have noticed anyway.
Set a coverage gate at 90% and the organisation will produce exactly 90%, by the cheapest route available. The cheapest route is assertion-free tests over the largest uncovered files, which is why coverage-gated codebases develop a characteristic test file: long, mechanical, calls everything, asserts almost nothing.
What coverage is genuinely good for is the complement. coverage report --show-missing on a module you just wrote tells you which branch you forgot, and that is a real, daily-useful signal. Display it, look at the diff coverage on a PR, talk about the uncovered lines — and do not make the number a threshold, because the moment it is a threshold it stops measuring anything.
If you want an automatable gate, gate on mutation score. It is the only widely available metric that asks whether a test would notice a bug, because it introduces the bug and checks.
Mocking what you do not own
responses.add(
responses.POST, "https://api.stripe.com/v1/charges",
json={"id": "ch_1", "status": "succeeded"}, status=200,
)That mock is not a fact about Stripe. It is your belief about Stripe, frozen at the moment you wrote it, and it will never be updated by anything except a failure in production.
When the belief is wrong — the field is paid not status, the error case returns 402 with a body you did not model, a required parameter was added — the mock keeps saying what you told it, and the suite stays green while the integration is broken. You have built a machine for converting your misunderstanding into confidence.
The standard resolution is a two-layer approach. Wrap the third party in a thin adapter that you own, expressed in your own domain types. Mock the adapter everywhere in the unit suite, freely and cheaply — it is yours, you control its contract, and mypy checks your fake against its protocol. Then have a small number of contract tests that exercise the adapter against the real thing, run on a schedule rather than on every commit, and whose failure means “our belief about their API expired”. That is the only test in the system that can detect an upstream change, and it needs to be exactly one place rather than three hundred mocks.
The vendor’s own sandbox or recorded-cassette tooling can serve the same purpose, with the same caveat: a cassette recorded in 2024 is a belief from 2024.
💡Your integration suite is slow, so someone proposes mocking the database as well — it is a dependency you did not write, after all. What is different about that case? click to reveal
The database is a dependency you did not write, but unlike a payment API it is available, deterministic, and free to run, and those three properties change the calculation completely.
The reason to mock an external HTTP service is that you cannot call it: it costs money, it rate-limits, it has side effects in someone else’s system, and it is not there at 3am when CI runs. None of that is true of Postgres in a container. You can start a real one in a few seconds, and every test then exercises the actual SQL, the actual constraints, the actual transaction semantics, and the actual type coercions.
What you lose by mocking it is enormous and specific. A mocked database cannot fail a NOT NULL constraint, cannot deadlock, cannot reject a duplicate key, cannot demonstrate that your migration and your model disagree, and cannot show you that the query returns Decimal where your code expected float. Those are the bugs the database layer actually has. A suite of tests against a fake ORM verifies that you can call methods on a fake ORM.
The speed problem is real and has a better answer: one session-scoped container, one transaction per test rolled back at teardown. That is milliseconds per test after a one-off startup, and it is the standard pattern — expensive-and-immutable at session scope, cheap-and-mutable per function.
The general rule: mock what you cannot run, not what you would rather not wait for. Slowness is an engineering problem with engineering solutions. Fidelity, once you have thrown it away, does not come back.
Flaky tests are a design signal
The instinctive response to a flaky test is a retry plugin, and it is the wrong response — not because retries are immoral, but because flakiness is almost always information about the system, and retrying deletes it.
Three causes cover nearly all of them:
Hidden shared state. A test passes alone and fails in the suite, or passes on one machine and fails when pytest-xdist reorders. Something is not being cleaned up, or a session-scoped fixture is being mutated, or there is a module-level cache. The test is telling you that two things which look independent are not — and that is exactly the property your production code has too, under concurrency.
Real time. Any test containing sleep, a timeout, or an assertion about elapsed time is a bet on scheduler behaviour. It will fail on a loaded runner. The fix is a controllable clock or an explicit synchronisation point, and the flakiness was telling you the code has no seam for time.
Real ordering. Iterating a set, relying on dict ordering across processes, or comparing unordered results with ==. This is usually a real bug in waiting: something downstream will eventually depend on the order.
In every case the flaky test found something before production did. A retry converts that discovery into noise, permanently, and the underlying defect ships. Quarantine a flaky test if you must — mark it, take it out of the gating run, file the ticket — but treat the quarantine list as a bug list, because that is what it is.
The one-paragraph version
Test the promises you made, at the boundary where you made them. Measure the suite by whether it would catch a bug, not by how many lines it touched. Own the seams you mock, and check the ones you do not own on a schedule. And when a test is unreliable, believe it.