Build Strings with join and Bytes with bytearray

Your script writes a CSV report, and it builds the text one line at a time. Save this as a script and run it:

import time

for rows in (50_000, 100_000, 200_000):
    lines = [f"item{i},{i % 7}\n" for i in range(rows)]
    start = time.perf_counter()
    text = ""
    for line in lines:
        text += line
    print(f"{rows:>7} rows: {time.perf_counter() - start:.2f}s")

On a laptop with Python 3.15, it printed:

  50000 rows: 0.20s
 100000 rows: 0.81s
 200000 rows: 3.29s

Each time the rows double, the loop takes about four times as long.

Every += copies the whole string

A string cannot change after Python creates it. So text += line cannot add to the old string. It makes a new string, copies all the old text into it, and then copies the line. Row 200,000 copies the 199,999 rows before it. Doubling the rows doubles the number of copies and also the length of each one, so the work grows four times. bytes objects work the same way.

Why the same loop can be fast

Put the loop inside a function, and it takes about 5 milliseconds for 200,000 rows. CPython has a shortcut. When a local variable holds the only reference to a string, += grows that string in place instead of copying it. The loop above runs at the top level of a script, where text is a global variable, so the shortcut does not apply.

Small changes to the code turn the shortcut off. Keep the text in an attribute or a list item, or keep a second name pointing at it, and the loop is slow again. Python implementations without reference counts, such as PyPy, do not have the shortcut. Write code that is fast without it.

Append the parts to a list, then join once

Append each piece to a list, and call "".join(parts) once at the end. join adds up the lengths, makes one string of the right size, and copies each piece into it once:

from collections.abc import Iterable


def number_lines(values: Iterable[int]) -> str:
    parts: list[str] = []
    for value in values:
        parts.append(f"{value}\n")
    return "".join(parts)


assert number_lines([10, 20]) == "10\n20\n"
assert number_lines([]) == ""

Build bytes in a bytearray

A bytearray is a version of bytes that you can change in place. It keeps spare room at the end, so most appends copy only the new bytes. Call bytes(buffer) once when you are done:

from collections.abc import Iterable


def encode_lines(lines: Iterable[str]) -> bytes:
    buffer = bytearray()
    for line in lines:
        buffer += line.encode("utf-8")
    return bytes(buffer)


assert encode_lines(["café\n"]) == "café\n".encode("utf-8")
assert encode_lines([]) == b""

Encode text with "utf-8". The "ascii" codec raises UnicodeEncodeError on a name like café. If the byte pieces already exist, b"".join(pieces) works too.

Joining a generator does not save memory

"".join(render(row) for row in rows) looks as if it handles one row at a time. It does not. join returns one complete string, and CPython first puts every generated piece into a list so it can add up their lengths.

Going further

PEP 8, the Python style guide, gives the same advice. It asks for "".join() where speed matters, because the += shortcut is specific to CPython.