Every team that adds automated quality checks eventually reaches the same fork. The tools can find far more than you can afford to block on. What do you do with the rest?
The two common answers are both bad.
Block on everything. The gate now fails for reasons that are sometimes wrong. People learn to re-run it, then to suppress it, then to ask for bypass permissions. Within a quarter the gate means “somebody was persistent” rather than “this code is correct.”.
Block on almost nothing. The gate means “it compiles”. The findings still exist, in a dashboard nobody opens.
The split
Run two things with different jobs, and be explicit about which is which.
The gate. Pass/fail. Blocks the merge. Small, and chosen so that a failure is essentially always a real problem:
- the formatter is satisfied,
-
the linter’s defect rules pass — undefined names, mutable defaults,
exceptcatching too much, a bareassertin production code, -
mypy --strictpasses, - the tests pass,
- the wheel builds and imports.
Every one of those has a near-zero false-positive rate. If a gate rule fires, the code is wrong.
The review score. A number, 0–100, shown on the pull request. Never blocks. Wide: complexity, function length, docstring coverage, comment density, test-to-code ratio, duplication, “this module has eleven imports and four responsibilities”. Many of these are wrong a third of the time, which is fine, because nothing depends on them being right.
The score’s job is to start a conversation, not to end one. It moves, people notice when it moves a lot, and nobody has to argue about whether a 47-line function is acceptable in this particular case at 6 p.m. on a Friday.
💡Which side does test coverage belong on? click to reveal
Mostly the score, with one narrow exception on the gate.
As a gate, a coverage threshold is a rule that can be satisfied without improving anything: a test that imports a module and asserts nothing raises the number. Worse, an 80% threshold on a PR that adds a well-tested feature to a poorly-tested codebase fails for reasons the author did not cause, which is the classic trust-spending false positive.
As a score it is genuinely informative — especially as a delta. “This PR drops coverage by four points” is a fact worth a comment; “coverage is 79.4%” is not.
The narrow gate exception: coverage must not decrease on the files this PR touched. That is attributable to the author, actionable without touching unrelated code, and almost never a false positive. It is a different rule from a global threshold, and it is the one worth blocking on.
Write the comment, not the rule code
A finding that says E722 teaches nobody anything. A finding that says this:
A bare
except:also catchesKeyboardInterruptandSystemExit, so your process can no longer be stopped cleanly —Ctrl-CandSIGTERMboth get swallowed. CatchExceptionif you mean “anything the code might raise”, or name the specific exceptions if you know them.
…teaches the reason, so the developer does not need the rule next time.
This is more work per rule, which is another argument for a small gate: you should be willing to hand-write a paragraph for every rule you block on. If the rule does not deserve a paragraph, it belongs in the score.
A useful discipline: the comment should say what goes wrong in production, not what the rule is called. “A mutable default is shared across every call” is a mechanism; “the second caller sees the first caller’s data” is the consequence, and the consequence is what people remember.
💡A rule on the gate turns out to fire wrongly about 5% of the time. It also catches a real bug about once a month. Keep it? click to reveal
Move it to the score, and this is worth being firm about even though the rule is doing real work.
Count the interactions rather than the bugs. A 5% false-positive rate on a team merging forty pull requests a week is two wrong blocks a week — roughly a hundred a year, against twelve real catches. Every one of those hundred teaches somebody that the gate is sometimes wrong, and that lesson generalises to the rules that are never wrong.
On the score, the rule keeps finding its twelve bugs a year: it appears on the PR, a reviewer looks, and when it is wrong they ignore it at no cost to anyone’s trust.
If you want it back on the gate, narrow it until the false positives stop — scope it to one package, tighten the pattern, require a second signal. A precise version of a noisy rule is usually available and is usually worth the afternoon.
The underlying asymmetry: a missed bug costs one incident, and a false positive costs a little bit of the gate’s credibility, forever.