Turn On pytest's Strict Checks

You mark a test as slow, with a typo:

import pytest


@pytest.mark.slwo
def test_big_report() -> None:
    assert sum(range(1000)) == 499500

Your nightly job runs pytest -m slow, so it never runs this test. pytest only prints a warning that is easy to miss:

PytestUnknownMarkWarning: Unknown pytest.mark.slwo - is this a typo?

With strict checks on, the same typo stops the run before any test starts:

ERROR collecting tests/test_report.py
'slwo' not found in `markers` configuration option

Turn on strict mode

Put this in pyproject.toml, or merge it into the [tool.pytest] table you already have:

[tool.pytest]
strict = true
addopts = ["-ra"]
testpaths = ["tests"]
markers = [
  "slow: tests that take longer to run",
  "integration: tests that talk to real services",
]

strict = true turns on the four checks in the table below.
markers lists every marker your tests may use, with a short description. pytest --markers prints the list.
-ra adds a summary line for every test that did not pass, including skips and expected failures, so you notice them.
testpaths tells pytest where to look when you run it without naming any tests.

Registering slow only declares the name. It does not skip anything. To leave the slow tests out, run pytest -m "not slow".

Setting	What it turns into an error
`strict_markers`	A marker that is not listed in `markers`.
`strict_config`	An unknown setting, such as `testpath` instead of `testpaths`.
`strict_xfail`	A test marked `xfail` that passes.
`strict_parametrization_ids`	Two parametrize cases with the same ID.

To turn one check off, set it to false, as in strict_parametrization_ids = false. The other three stay on. Leave a comment that says why.

Check your understanding You register slow in markers. Does pytest now skip the slow tests? Show answer

No. Registering a marker only tells pytest the name is real, so strict mode accepts it. Every marked test still runs. To leave them out, select by marker with pytest -m "not slow".

Let a fixed bug fail the run

xfail marks a test you expect to fail, for example a test for a bug that is still open:

import pytest


@pytest.mark.xfail(reason="rounding bug, issue 42", raises=AssertionError)
def test_rounding() -> None:
    assert round(2.675, 2) == 2.68

With strict_xfail, a test like this that starts to pass fails the run:

FAILED tests/test_money.py::test_rounding - [XPASS(strict)] rounding bug, issue 42

That is what you want. The test passes because someone fixed the bug, so delete the marker and keep the test. Without strict, the run stays green, and the old marker would also hide the bug if it came back.

raises=AssertionError keeps the expected failure narrow. Only a failed assertion counts as expected. Any other exception fails the test as usual.

Check which config file pytest read

pytest reads one config file and ignores the others. If a folder has both pytest.toml and pyproject.toml, pytest.toml wins, even when it is empty. So a stray empty pytest.toml turns off every setting in pyproject.toml. The header of each run names the file pytest used:

rootdir: /home/you/shop
configfile: pytest.toml (WARNING: ignoring pytest config in pyproject.toml!)

Inside pyproject.toml, use either [tool.pytest] or the older [tool.pytest.ini_options]. pytest stops with an error if the file has both.

Turn warnings into errors separately

strict does not turn warnings into errors. Add filterwarnings for that:

[tool.pytest]
filterwarnings = [
  "error",
  # Remove after the upgrade to parser 3.
  "ignore:legacy parser is deprecated:DeprecationWarning",
]

"error" makes any warning fail the test that raised it, so you see a deprecation before the library removes the feature. The second line ignores one known warning, matched by the start of its message and its category. When a warning matches several lines, the last one wins. Keep each ignore this narrow, and say when to remove it.

Going further

pytest 9.0 added the strict setting and the [tool.pytest] table. On older versions, use [tool.pytest.ini_options] and turn the checks on one by one, with addopts = ["--strict-markers", "--strict-config"] and xfail_strict = true. pytest 9 still accepts xfail_strict as another name for strict_xfail.

strict may turn on more checks in later pytest versions, so read the release notes when you upgrade.

A standalone pytest.toml uses a [pytest] table. In each folder, pytest looks for pytest.toml, .pytest.toml, pytest.ini, .pytest.ini, pyproject.toml, tox.ini and setup.cfg, in that order. The last three count only if they have a pytest section. Pass -c pyproject.toml to choose a file yourself.