You want to know how much memory a cache of a thousand entries uses, and
you try sys.getsizeof:
import sys
items = [object() for _ in range(1000)]
shallow = sys.getsizeof(items)
total = shallow + sum(sys.getsizeof(item) for item in items)
print(shallow, total) # 8856 24856 on 64-bit CPython 3.12 to 3.15
sys.getsizeof returns the shallow size, the memory of the object
itself, not of the objects it points at. For a list, that is the list's own
header and its array of pointers to the items. The thousand objects are
extra, so you have to visit them and add their sizes yourself. The numbers
change with the Python version and the platform.
Count each object once
Two lists can print the same and still use different amounts of memory:
shared = [[1, 2, 3]] * 3
distinct = [[1, 2, 3] for _ in range(3)]
print(shared[0] is shared[1]) # True, one inner list three times
print(distinct[0] == distinct[1]) # True
print(distinct[0] is distinct[1]) # False, three separate inner lists
shared holds one inner list three times, so that list's memory counts
once. distinct holds three equal lists, and each one counts. So tell
objects apart by id, not by ==. Keep a set of the ids you have already counted, and
skip an object whose id is in it.
The same set handles a list that contains itself. The second time you
reach the list, its id is already in the set, and you stop.
A list keeps spare room
A CPython list reserves room for future appends, so its shallow size grows in steps, not by one slot per item:
import sys
values: list[int] = []
previous = sys.getsizeof(values)
for value in range(40):
values.append(value)
current = sys.getsizeof(values)
if current != previous:
print(len(values), current)
previous = current
The spare room makes appending fast. It also means two lists of the same length can report different sizes, depending on how you built them.
Check your understanding Why can multiplying one cache entry's size by the number of entries mislead you? Show answer Hide answer
A shallow size leaves out everything the entry points at. A deep size of one entry can count objects that every entry shares, so multiplying counts them again for each entry. Entries also differ in size. Measure the whole cache in one pass that counts each object once.
Choose what to follow
A full answer would follow every reference Python knows about, which is far more than you usually want. The exercise follows a small, clear set:
- the keys and values of a mapping, such as a
dict - the items of a sequence or a set, such as a
list, atupleor aset
It stops at str, bytes and bytearray. Their shallow size already
includes their characters or bytes. Looping over a string gives you
one-character strings, which are not part of its memory.
It also stops at everything else, such as class instances, functions and generators. An instance's attributes, for example, are outside this estimate.
Check your understanding
Why check isinstance(obj, Sequence) from collections.abc instead of isinstance(obj, list)?
Show answer
Hide answer
The abstract class also matches tuple, collections.deque and your own
sequence classes, so one rule covers them all. A check for list would
miss them, and you would add one branch per type.
The total is an estimate
Even a careful total is not the memory the operating system reports for
your process. The process also holds the interpreter, modules, and memory
that Python freed but kept for reuse. Deleting the root object can also
free less than your total, because other code may still use some of the
objects. To see where memory grows over time, use tracemalloc, which the
next article teaches.
Going further
The sys.getsizeof documentation
explains how a class can report its own size with __sizeof__. An object
from a C extension, such as a NumPy array, decides for itself whether to
include its data buffer.