PEP 686: UTF-8 by default, and the migration hazard

This is the most common cross-platform data-corruption bug in Python, and it is invisible on the developer’s machine.

text = open(path).read()          # which encoding is that?

Until 3.15, the answer is locale.getencoding(). On your Linux laptop and in your Linux CI that is UTF-8. On a Windows runner in a European locale it is cp1252. The same file, the same code, two different strings — and when the input happens to be ASCII, which it is in every fixture anyone wrote by hand, the two agree perfectly. The bug ships the first time a customer’s name has an accent in it.

What changes, and when

PEP 686 makes UTF-8 mode the default in 3.15: open(), TextIOWrapper and friends use UTF-8 regardless of locale, with PYTHONUTF8=0 as the opt-out for code that genuinely needs the old behaviour.

The migration tool has existed since 3.10. Run with -X warn_default_encoding (or PYTHONWARNDEFAULTENCODING=1) and every implicit-encoding call raises an EncodingWarning pointing at the call site, not at the library. Turn it into an error with -W error::EncodingWarning and you have a fitness function.

And if you genuinely want the locale’s encoding, say so: encoding="locale" (3.10+) is explicit, greppable, and does not trigger the warning.

Correct practice today, on 3.12: always pass encoding=. Not because 3.15 is coming, but because the code has to run on 3.12 through 3.15 and behave the same on all of them.

BOMs, briefly

A byte-order mark is a U+FEFF at the start of a file. Three facts:

• utf-8-sig strips a UTF-8 BOM; plain utf-8 decodes it into a literal \ufeff character at the head of your string, which then fails every startswith you write.
• The utf-16 codec (no -le/-be suffix) reads the BOM to decide byte order, and consumes it.
• A BOM is a hint, not a declaration. If the file’s own metadata says something else, the metadata wins — silently overriding it is how a mojibake bug becomes unreproducible.

What you are building

def sniff_encoding(raw: bytes) -> str
def decode_config(raw: bytes, declared: str | None) -> tuple[str, str]
def solve(raw: bytes, declared: str | None) -> dict[str, str]

sniff_encoding returns "utf-8-sig" for a UTF-8 BOM, "utf-16" for either UTF-16 BOM, and "utf-8" otherwise.

decode_config uses declared if given, else the sniffed encoding, and returns (text, encoding_used). On failure it raises ConfigEncodingError (given; it carries the attempted encoding) chained from whatever the codec raised — raise ... from exc.

solve returns {"status": "ok", "text": ..., "encoding": ..., "cause": ""} or {"status": "error", "text": "", "encoding": ..., "cause": <exception class name>}. The cause is read off __cause__, so an unchained raise fails the tests: the chain is the requirement, not decoration. It is also what turns a 2 a.m. traceback from “config failed to load” into “config failed to load, because byte 3 is not valid UTF-8”.

Note the two distinct failure modes: bad bytes give UnicodeDecodeError; a bad codec name gives LookupError. Catch both, or a typo’d encoding in someone’s config file becomes an unhandled exception at import time.

The typing detail

mypy 2.0 turned on --strict-bytes by default (the PEP 688 alignment): bytearray and memoryview are no longer implicitly assignable to bytes. Code that read from a socket into a bytearray and passed it to a (raw: bytes) function used to type-check and now does not. The fix is almost always to widen the parameter to collections.abc.Buffer or to call bytes() explicitly — not to loosen the annotation to Any.