Three tools, three different contracts, and the differences bite in production rather than in the tutorial.
| Expression | Missing key | Mutates? |
|---|---|---|
d.get(k) |
returns None |
no |
d.get(k, default) |
returns default |
no |
d.setdefault(k, default) |
inserts default, returns it |
yes |
defaultdict(f)[k] |
calls f(), inserts, returns it |
yes |
defaultdict mutates on read
This is the whole lesson. d[k] on a defaultdict is not a query, it is a
write:
from collections import defaultdict
seen = defaultdict(list)
if seen["alice"]: # looks like a membership test
...
len(seen) # 1 -- "alice" now exists, with an empty list
In a long-lived service this is a slow memory leak with an innocuous
fingerprint: the dict grows one entry per lookup, and every entry is an
empty container. The profile shows a dict of empties and no obvious writer.
Worse, the check if seen["alice"]: reads as a question and behaves as a
statement, so a reviewer skims past it.
setdefault has exactly the same insert-on-call behaviour, but at least it
looks like a method call, and it takes the default by value:
d.setdefault(k, expensive()) # `expensive()` runs on EVERY call
The default expression is evaluated eagerly whether or not it is needed. If
it is expensive, or has side effects, setdefault is the wrong tool —
defaultdict takes a factory and calls it only on a miss.
💡You are counting with defaultdict(int) and want to log the
click to reveal
keys that never appeared. Why does for k in expected: if counts[k] == 0 not
work, and what is the fix?
Every counts[k] inserts 0 for a key that was missing, so by the end of the
loop counts contains an entry for every key in expected and the “which
keys are missing” information has been destroyed by the act of asking.
The fix is to read without the default machinery:
missing = [k for k in expected if k not in counts]
# or
missing = [k for k in expected if counts.get(k, 0) == 0]
in and .get() never insert. The general rule: defaultdict is for the
write path. Any read path — a report, a validation, a log line — should use
.get() or in.
Do not let it escape
The behaviour is a property of the object, not of the scope it was built in.
If a public function returns the defaultdict it accumulated into, every
caller inherits insert-on-read — including callers who reasonably assume a
dict raises KeyError. A test that asserts KeyError on an absent key will
fail, an API that serialises the mapping will see it grow while it iterates,
and the type annotation -> dict[str, list[int]] will not warn you, because
defaultdict is a dict.
Convert on the way out:
def group_by_key(rows: list[tuple[str, int]]) -> dict[str, list[int]]:
grouped: defaultdict[str, list[int]] = defaultdict(list)
for key, value in rows:
grouped[key].append(value)
return dict(grouped) # <-- the important line
dict(grouped) is a shallow copy: cheap, and the value lists are shared, not
duplicated. Use defaultdict internally, hand back a dict.
The annotation --strict forces on you
grouped = defaultdict(list)
# error: Need type annotation for "grouped"
This looks like the type checker being obtuse and is actually the most useful
error in the module. defaultdict(list) tells mypy the value is some list
and nothing about the element type or the key type. Writing
defaultdict[str, list[int]] forces you to state the shape of the grouping —
which is exactly the fact a reader of the function needs and would otherwise
have to reconstruct from the loop body.
Note that defaultdict[str, list[int]] as a subscript works at runtime from
3.9 onward, so no typing.DefaultDict import is needed.
💡d.setdefault(k, []).append(x) is a very common idiom. When is
click to reveal
it fine, and when should it be a defaultdict?
It is fine — and arguably clearer — when the grouping is local, small, and
built in one place. It creates a fresh [] on every call, which is wasted
work on a hit but harmless at small scale, and it keeps the container a plain
dict so nothing leaks to callers.
Prefer defaultdict when the default is expensive to construct (it is only
called on a miss), when the same grouping happens in several places (the
factory states the intent once), or when the hot loop is long enough that
allocating a throwaway list per iteration shows up in a profile.
Prefer neither when the value type is immutable: counts[k] = counts.get(k, 0) + 1
is perfectly readable, and Counter is better still.