Seven mechanisms for testing structure

Behaviour is easy to test: call the function, compare the result. Structure is not. “The domain layer must not depend on the database”, “this API must make misuse impossible”, “importing this package must be cheap” — none of those is a return value.

They are all testable anyway. Seven mechanisms cover essentially everything, and once you have them you can build the gates your own team needs rather than waiting for a tool to ship them.

M1 — Collaborator as parameter, plus an adversarial fake

Pass the dependency in; supply a fake that is deliberately hostile. Not a polite stub that returns canned data — a fake that supplies information the real dependency cannot, or misbehaves in a way the real one rarely does.

A clock you can set to exactly the window boundary. A store that returns None for a cold key on the first call and a stale value on the second. An AuditSink that raises on its third call and succeeds on the fourth. The real dependency does all of those eventually, at 3 a.m., once.

This is the mechanism that makes dependency injection worth its ceremony: the hostile fake is a test you cannot write against a hard-coded collaborator.

M2 — Environment sabotage

A fresh subprocess with the world removed:

code = (
    "import sys, os\n"
    "sys.modules['sqlite3'] = None\n"
    "os.environ = {}\n"
    "import pricing.domain\n"
    "print(pricing.domain.total(order))\n"
)
subprocess.run([sys.executable, "-P", "-c", code], check=True)

This is the strongest structural test there is, because it cannot be fooled by where an import is written. A function-local import sqlite3 is invisible to a static graph tool and fatal here. Note it asserts the layer computes, not merely imports — importing successfully proves less than you would like.

M3 — Import-graph fitness

An import-linter contract (layers, forbidden, independence), or ruff analyze graph plus your own assertion. Transitive, which is the whole point: domain → helpers → sqlite3 is a violation no name-matching rule will find.

M4 — Module-namespace assertions

assert set(pricing.__all__) == {"quote", "Money", "Order"}
assert "sqlite3" not in sys.modules            # after `import pricing`

Cheap, and they catch two very different regressions: something became public by accident, and something heavy started loading at import. Pair the second with python -X importtime as an import-cost budget.

M5 — Negative type assertions

The highest-leverage mechanism for API design, and the least known. Because --strict includes --warn-unused-ignores, an unnecessary # type: ignore is itself an error:

checksum_of(begin("a"))            # type: ignore[arg-type]
add_chunk(seal(begin("a")), b"")   # type: ignore[arg-type]

If the design permits either call, the ignore is unused, mypy errors, and the gate fails. This converts “did you design the API so that misuse is impossible?” into a boolean. It is exactly how typeshed asserts negative cases, and it is the only automated way to test that something is not expressible.

M6 — AST fitness functions

When the property is about how the code is written — no set_start_method anywhere, no pool constructed at import time, no bare except: — walk the AST and assert. Reach for this last: it is the easiest to get subtly wrong, and a check that is wrong 5% of the time costs more trust than it earns.

M7 — Differential and metamorphic testing

Run the same forty cases through two implementations of the same Protocol — the SQLite adapter and the in-memory one — and assert they agree.

This is the one that tests whether your abstraction is honest. Every other mechanism checks shape, and shape can be perfect while the fake is simply easier to satisfy than reality: different rounding, different null handling, different ordering, a different answer for a missing key. Tests pass against the fake and production does something else. A differential test is the only thing on this list that catches that.

💡You want to prove a library has no import-time side effects. Which mechanisms apply, and what does each miss on its own? click to reveal

M2 and M6, and they are complementary in a way worth understanding.

M6 (the AST check) names the line. It tells you ThreadPoolExecutor is constructed at line 14 and that a default argument at line 22 runs at import. What it misses is anything it was not written to look for — a side effect achieved through a helper, through a decorator, through a metaclass, or through an import of another module that does the work.

M2 (the subprocess) catches the effect regardless of how it was written: a thread was started, a module was loaded, the environment was read. What it misses is the location — it tells you something happened, not which line did it.

Run both. The subprocess test is the one that must never be removed, because it is the one that cannot be routed around; the AST check is the one that makes the failure diagnosable in ten seconds instead of twenty minutes.

What makes a bad structural problem

Three anti-patterns, all of which look like they are testing design and are not:

Anything gradeable only by string-matching the source. “Your solution must contain the word Protocol“ tests spelling. The learner can satisfy it without understanding it, and cannot fail it while understanding it perfectly and using an ABC for a good reason.

“Return the correct module layout as a nested dict.” That tests recall of a convention, not the ability to place a seam. There is no design decision in it — the answer was given in the question.

“Implement the Observer pattern”, where the structure was dictated. If the problem statement specifies the classes, the methods and the relationships, the learner is transcribing, not designing. The interesting question — should this be an Observer at all? — was answered before they started.

What must stay human

Some things are genuinely judgment and do not have a machine-checkable answer:

• Naming. No metric distinguishes process_data from normalise_totals.
• Whether to abstract at all. The right number of layers for a 400-line tool is one. A checker that rewards structure will reward it there too.
• Whether this seam is the seam that will matter. That is a prediction about which axis will change, and being right about it is experience.

For these, ship an explanation and an interactive illustration where one helps — and stop. Do not fake-automate taste. A metric that pretends to score naming is worse than no metric, because it is confidently wrong and it teaches people to write for the metric.