There is a memory optimisation here that is real, measurable and worth using, and a correctness belief attached to it that is a time bomb. They are easy to confuse because they are the same mechanism.
The optimisation is real
Parse ten million log lines, each with a level field drawn from five distinct values. Naively you now hold ten million string objects, of which 9,999,995 are duplicates. Each is a PyObject with a header, a hash cache, and a payload — on the order of 50 bytes for a short string. That is roughly half a gigabyte to store five distinct words.
import sys
level = sys.intern(fields[2])
sys.intern looks the string up in a global table and returns the canonical object if one exists, adding it if not. Now you hold ten million references to five objects. The pointer array is unavoidable; the 500 MB of duplicate payloads is gone.
This is not a niche trick — it is the same idea as dictionary encoding in pandas’ Categorical and Arrow’s DictionaryArray, and it is why those types exist. When a column has low cardinality relative to its length, storing indices into a dictionary of distinct values beats storing the values.
The correctness belief is not
What people then conclude is that sys.intern(a) is sys.intern(b) can be used to compare, or worse, that string literals are automatically interned so x is "active" is safe. Article 11.7 covers the second. The first is true but pointless — == on two interned strings already short-circuits on identity, so you gain nothing and you have coupled your correctness to a caching layer.
The deeper issue is what the language actually promises. The −5..256 small-integer cache and automatic literal interning are, verifiably, not documented anywhere in the language reference — not in stdtypes, not in datamodel, not in expressions, not in the C API documentation for integers. sys.intern itself is documented; what CPython chooses to intern for you is not. That absence is the specification declining to promise, and it is the whole lesson.
The free-threading trap
sys.intern‘s documentation carries a warning that most readers skim: “Interned strings are not immortal; you must keep a reference to the return value.” That is the normal build. Intern a string, drop every reference, and it leaves the table.
On the free-threaded build this is inverted: “any interned string will become immortal, surviving until interpreter shutdown.” This is not an oversight — immortality is how you avoid atomic refcount traffic on a table shared by every thread — but it changes the risk profile of the same line of code completely.
level = sys.intern(record["level"]) # fine for five known values
user_agent = sys.intern(record["ua"]) # unbounded on the free-threaded buildInterning a value drawn from a small, closed set is a memory win on both builds. Interning a value that comes from untrusted input — a header, a user-supplied tag, a URL path — is a bounded-size cache on the GIL build and an unbounded leak that cannot be reclaimed on the free-threaded build. An attacker who can send you a million distinct values has permanently allocated a million strings in your process.
💡You are adding sys.intern to a hot parser. What would you need to establish about the field before interning it, and what would you do instead if you could not establish it?
click to reveal
You need to bound the cardinality, and bound it by something other than optimism. “Levels are one of five” is a schema fact you can point at. “User agents are usually one of a few dozen” is a hope about traffic, and traffic is exactly what an adversary controls.
So: intern the field if the set of possible values is closed and defined somewhere — an enum, a protocol spec, a database enum column, a documented header value. Do not intern a field whose value space is “whatever the client sent”.
If you cannot bound it and still want the deduplication, use a cache you control rather than the interpreter’s global table: a dict[str, str] with an LRU bound, or a WeakValueDictionary so entries disappear when nothing references them. Both give you the dedup and neither is permanent. That is strictly more code and strictly more predictable, which is the usual trade when moving from an interpreter feature to an explicit one.
The typed alternative, which is better anyway
If the value really is drawn from a small closed set, the interning question is the wrong question. Model the set:
from enum import StrEnum
from typing import Literal
class Level(StrEnum):
DEBUG = "debug"
INFO = "info"
ERROR = "error"
Status = Literal["active", "suspended", "closed"]
Both give you what interning gives you — one object per distinct value — and both give you something interning cannot: the checker rejects a typo. Level("warn") raises at the parse boundary rather than silently taking the wrong branch three hundred lines later. Literal costs nothing at runtime at all; the values are the same interned literals, and --strict will not let you compare a Status against "activ".
Note the 3.11 behaviour change that catches people migrating to StrEnum: str(Level.INFO) gives "info", because StrEnum and IntEnum inherit __str__ from their mixed-in type. On 3.10 the equivalent class Level(str, Enum) rendered as "Level.INFO". Any log line, cache key or serialised field that interpolated one of these changed shape on that upgrade, silently.
What to take away
Interning is a memory tool. It has one correct use — reduce duplicate payloads when cardinality is low and bounded — and one incorrect use, which is to reason about identity.
When you catch yourself thinking “these two strings will be the same object”, stop and ask what you actually need. If you need equality, use ==. If you need a closed set, use Literal or StrEnum. If you need memory, use sys.intern on a field whose cardinality you can name, or a bounded cache of your own on one you cannot.
💡sys.intern returns the canonical object. Explain why the documentation insists you keep a reference to the *return value* rather than to the string you passed in.
click to reveal
Because on the normal build the interned string is kept alive by an ordinary reference from the global table, and that table holds a borrowed reference — the entry does not itself keep the object alive. When the last real reference goes away, the object is deallocated and the entry is removed.
The string you passed in may not be the object you got back. If "info" was already interned, sys.intern(parsed) returns the existing canonical object and your parsed string is a separate object that will be freed. Holding onto parsed keeps the wrong object alive and does nothing to preserve the canonical one; holding onto the return value is what participates in the table’s lifetime.
In practice this means s = sys.intern(s) — rebinding — is the correct spelling, and sys.intern(s) as a bare statement is a no-op with a cost.