Bytes, Text and Line Endings

A user signs up with the name café. Your code saves it to a file, reads it back as text and as bytes, and gets two different lengths:

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as folder:
    path = Path(folder) / "name.txt"
    path.write_text("café", encoding="utf-8")
    text = path.read_text(encoding="utf-8")
    raw = path.read_bytes()
    print(len(text), len(raw))  # 4 5
    print(raw)                  # b'caf\xc3\xa9'

Both numbers are right. They count different things.

A file holds bytes, and a str holds characters

Every file, socket and pipe holds bytes, numbers from 0 to 255. A str holds Unicode characters, which Python calls code points. An encoding is the rule that turns one into the other. UTF-8, the most common one, uses one byte each for c, a and f, and two bytes for é. So "café" is 4 characters and 5 bytes.

encode turns text into bytes, and decode turns bytes back into text:

data = "café".encode("utf-8")
print(data)                  # b'caf\xc3\xa9'
print(data.decode("utf-8"))  # café

Opening a file in text mode decodes for you. read_bytes(), and open with mode "rb", give you the bytes as they are.

Name the encoding every time

open() and Path.read_text() take an encoding argument. Leave it out, and Python 3.12 to 3.14 use the operating system's locale encoding. On a Windows machine set to English, that is usually cp1252, not UTF-8, so the same file reads differently there. Pass encoding="utf-8" every time, and your code reads files the same way on every machine.

Bytes that are not valid UTF-8 raise UnicodeDecodeError. These bytes come from Latin-1, an older encoding that stores é as one byte:

latin = "café".encode("latin-1")
print(latin)  # b'caf\xe9'
try:
    latin.decode("utf-8")
except UnicodeDecodeError as error:
    print(error)
# 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data
print(latin.decode("utf-8", errors="replace"))  # caf�

errors="replace" puts � where the bytes are not valid UTF-8. That is fine for showing text to a person. For data you store, compare or sign, reject the input instead, because the replaced text is no longer what was sent.

Text mode changes line endings

Windows ends each line with two characters, \r\n. Linux and macOS use \n. When you read in text mode, Python turns \r\n, and a lone \r, into \n, so your code has to handle only \n. Pass newline="" to keep the line endings exactly as they are in the file:

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as folder:
    path = Path(folder) / "rows.txt"
    path.write_bytes(b"A\r\nB")
    with path.open(encoding="utf-8") as handle:
        print(repr(handle.read()))  # 'A\nB'
    with path.open(encoding="utf-8", newline="") as handle:
        print(repr(handle.read()))  # 'A\r\nB'

Writing works the other way round. In text mode, each \n you write becomes the platform's line ending, which is \r\n on Windows. The csv module handles line endings itself, so open CSV files with newline="", as its documentation says.

Common mistakes

• Checking a byte limit with len() on a str. A database column that holds 20 bytes does not hold every 20-character name. Count len(name.encode("utf-8")) when the limit is in bytes.
• Calling str() on bytes. str(b"caf\xc3\xa9") gives the text b'caf\xc3\xa9', with the b and the quotes. Call .decode("utf-8").

Going further

• Python 3.15 turns on UTF-8 mode by default, so open() without an encoding uses UTF-8 there (PEP 686). In older versions, python -X utf8 or the environment variable PYTHONUTF8=1 does the same.
• len() counts code points, which is not always what a reader sees as one character. é can also be written as e followed by a combining accent, which is two code points. unicodedata.normalize("NFC", text) joins them into one.
• Read the Python documentation on text encoding and on newline handling in open().