Compare strings with ==. Two equal strings are often two separate
objects:
a = "app:info"
b = "".join(["app:", "info"]) # built at run time
print(a == b) # True
print(a is b) # False
Interning is when Python reuses one object for equal strings. CPython
interns the names in your code, and literals that look like names, such as
"info". So an is check against a literal can print True in the REPL.
Text you read from a file, a socket or input() is usually a new object,
and the same check then fails in production. == compares the characters,
so it works either way.
Intern on purpose to save memory
A log parser reads a million lines. Each line starts with a level such as
info or error, and each level it splits off is a new string object. A
million copies of five words waste memory. sys.intern returns one shared
object for each distinct value:
import sys
lines = ["info start", "error disk", "info done"]
levels = [sys.intern(line.split()[0]) for line in lines]
print(levels) # ['info', 'error', 'info']
print(levels[0] is levels[2]) # True, one object for both "info" strings
Keep the string that sys.intern returns, and stop using the one you
passed in. The call does not change your original string, or any other
name that points at it.
Interning costs a lookup for every string. The saving depends on how many values repeat and how long you keep them. Measure memory with and without it before you keep it.
Intern only a small, known set of values
In the standard build, an interned string stays in memory while your code still uses it, like any other string. In the free-threaded build, the optional build of Python without the GIL, an interned string stays until the program exits.
So don't intern text that can take any value, such as request paths or tags typed by users. On a free-threaded server, every new path would add memory that is never given back. Intern a field only when its values come from a small set you know, such as log levels. Check each value against that set first.
When you know every value in advance, a StrEnum does the check for you.
Each member is a single object, so you get the reuse as well:
from enum import StrEnum
class Level(StrEnum):
INFO = "info"
ERROR = "error"
level = Level("info")
print(level is Level.INFO) # True
print(level == "info") # True, a StrEnum member is also a str
try:
Level("warn")
except ValueError as err:
print(err) # 'warn' is not a valid Level
A Literal type tells the type checker which strings a function accepts:
from typing import Literal
type LevelName = Literal["info", "error"]
def log(level: LevelName, message: str) -> None:
print(f"[{level}] {message}")
log("info", "started")
mypy rejects log("warn", "started"), but nothing checks the value when
the program runs. Text from outside your program still needs a runtime
check, such as Level(text).
Check your understanding
A teammate writes if level is "error": and says it is faster than ==. Should you accept it?
Show answer
Hide answer
No. The check is wrong whenever level is a separate object with the same
text, which is common for text read from input. Speed is not a reason
either. When two strings are the same object, == notices that first and
returns at once.
Check your understanding Your web service interns every request path to save memory. What happens over a week? Show answer Hide answer
Each new path adds an interned string. On the free-threaded build an interned string stays until the program exits, so memory grows with every distinct path. On the standard build an interned path stays only while something uses it, and a path used once saves nothing. Intern only values from a small, known set, and check them first.
Going further
- If values repeat a lot but the set is open, a bounded cache such as
functools.lru_cache(maxsize=1024)reuses recent strings and forgets old ones.maxsizecounts entries, not bytes. - The
sys.interndocumentation notes that interning can also speed up dictionary lookups. - The free-threading guide lists the objects that the free-threaded build keeps until the program exits.