itertools.groupby is a run-length encoder, not a SQL GROUP BY. It
walks the input once and starts a new group every time the key changes from
the previous element. That is the whole implementation, and both of its
famous failure modes fall directly out of it.
Failure one: it does not sort
data = ["apple", "avocado", "banana", "apricot"]
[(k, list(g)) for k, g in groupby(data, key=lambda s: s[0])]
# [('a', ['apple', 'avocado']), ('b', ['banana']), ('a', ['apricot'])]
Three groups for two distinct keys, and 'a' appears twice. Nothing raises.
If you build a dict from that, the second 'a' group silently overwrites the
first and you lose two records with no error and no log line.
The fix is one word, and it is a real cost: groupby(sorted(data, key=f), key=f).
Sorting is $O(n \log n)$ and materialises the whole input, which cancels most
of the laziness groupby gave you. If your data is genuinely too large to
sort, groupby is the wrong tool — use a defaultdict accumulation, which is
$O(n)$ and needs no ordering.
Use the same key function for both. Passing sorted(data) (no key) and
groupby(data, key=f) is a common near-miss that works on your fixture and
breaks on real data.
Failure two: the group iterator is invalidated
The group handed to you is a view over the shared underlying iterator. The
moment you advance the outer groupby, the previous group is spent:
list(groupby(["a", "b"], key=lambda s: s))
# [('a', <itertools._grouper>), ('b', <itertools._grouper>)]
[(k, list(g)) for k, g in list(groupby(data, key=f))]
# [('a', []), ('b', [])] <-- right keys, no data
This is the dangerous one, because a test that asserts on the keys passes
while every record has vanished. Anything that walks the outer iterator ahead
of you triggers it: list(), sorted(), reversed(), storing the pairs to
process later, or a nested loop that consumes lazily.
The rule is: consume each group before you ask for the next one. In
practice that means list(group) (or sum(...), len(...) — anything that
drains it) inside the loop body.
💡Why is {k: list(g) for k, g in groupby(sorted(data), key=f)}
click to reveal
fine, while dict(groupby(sorted(data), key=f)) is not?
The comprehension calls list(g) while g is still the current group. The
loop machinery only advances the outer iterator when the body finishes, so
each group is drained at the moment it is valid.
dict(...) builds the mapping by consuming the whole outer iterator first,
storing the _grouper objects as values. Each advance invalidates the
previous group, so by the time you look at any of them they are all exhausted.
You get a dict with the right keys and <itertools._grouper> objects as
values that yield nothing.
The general shape: any function that materialises the pairs without touching the groups is wrong, and it is wrong silently.
Typing
groupby(rows, key=lambda row: row[0])
With a lambda, mypy usually cannot infer a useful key type and the group key
degrades to Any — which then propagates into whatever you index with it,
disabling checking on the very dict you are building. Annotate the key
function:
def by_name(row: tuple[str, int]) -> str:
return row[0]
for name, group in groupby(sorted(rows, key=by_name), key=by_name):
... # name is str
A named key function also means you literally cannot pass different keys to
sorted and groupby, which removes failure one by construction. Two
problems, one refactor.
💡When should you use groupby at all, given the sort it needs?
click to reveal
Three situations.
The data is already ordered. A sorted database result, a time series, a
sorted log file, the output of a merge. Here groupby is $O(n)$ and streams —
you can group a 100 GB sorted file in constant memory, which no
defaultdict accumulation can do.
You want the runs, not the groups. Compressing [1,1,1,2,2,1] to
[(1,3),(2,2),(1,1)], detecting state changes in a signal, collapsing
consecutive duplicate log lines. Here the “bug” is the feature.
You need to sort anyway. If the output has to be in key order, you were
paying for the sort regardless, and groupby is then cheaper and clearer than
a dict accumulation followed by a sort of the keys.
Outside those, a defaultdict(list) accumulation is $O(n)$, needs no
precondition, and cannot be invalidated.