Decode a Config File with the Right Encoding

A colleague edits settings.ini on Windows, in an editor that saves text as cp1252, a common Windows encoding for Western European languages. Your service reads the file on Linux and stops with this error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data

A file holds bytes. An encoding is the rule that turns text into bytes and back, and you must decode with the encoding the editor saved the file in. The word café is the bytes b"caf\xe9" in cp1252, and b"caf\xc3\xa9" in UTF-8:

b"caf\xe9".decode("cp1252")   # 'café'
b"caf\xe9".decode("utf-8")    # UnicodeDecodeError

A wrong guess can also succeed and give you garbled text. So choose the encoding on purpose, and report which one you used. In this exercise the bytes are already in memory.

Choose the encoding

Some editors start a file with a byte-order mark (BOM), a few marker bytes that name the file's encoding. The codecs module has them as constants. Write sniff_encoding(raw: bytes) -> str, which checks how raw starts:

The utf-8-sig encoding removes the BOM from the text. Plain utf-8 would keep it as the invisible character U+FEFF. The utf-16 encoding reads the BOM to learn the byte order, and removes it too.

Decode, and keep the original error

Write decode_config(raw, declared) -> tuple[str, str]. declared is the encoding that a header or a setting names, or None if nothing names one.

1. If declared is not None, use it exactly as given, even when it is an empty string. Otherwise call sniff_encoding(raw).
2. Decode raw with that encoding and the default strict error handling. Don't replace bad bytes, and don't retry with another encoding.
3. Return (text, encoding), with the name exactly as you chose it. A declared "latin-1" comes back as "latin-1", not as another spelling of the same encoding.
4. Decoding raises UnicodeDecodeError for bytes that don't fit the encoding, and LookupError for an unknown encoding name. Catch only those two, and raise the supplied ConfigEncodingError(encoding) with raise ... from exc.

raise ... from exc stores the original error in the new error's __cause__ attribute, and the traceback shows both errors. So the caller learns which encoding failed, and why.

In this exercise, a declared encoding wins over a BOM, even when it is wrong. UTF-8 bytes with a BOM, declared as latin-1, decode without an error, and the text starts with . A decode that succeeds does not prove that the encoding was right.

Return the result

Write solve(raw, declared) -> dict[str, str], which returns exactly these keys:

• On success, {"status": "ok", "text": text, "encoding": encoding, "cause": ""}.
• On ConfigEncodingError, {"status": "error", "text": "", "encoding": exc.encoding, "cause": type(exc.__cause__).__name__}.

For b"caf\xe9" declared as "cp1252", the text is café. With declared set to None, UTF-8 fails, and cause is "UnicodeDecodeError".

Edge cases

• Empty raw with nothing declared is valid UTF-8, and gives empty text.
• "", "locale" and made-up names all raise LookupError in bytes.decode.
• UTF-16 uses two bytes per unit, so an odd number of bytes declared as "utf-16" raises UnicodeDecodeError.

Going further

open(path) without encoding= picks a default, and the default depends on the Python version and the system. Write Code That Runs on Several Python Versions explains the difference. Pass encoding= every time, and run Python with -X warn_default_encoding to get an EncodingWarning for each call that leaves it out.