pathlib has moved faster in the last four releases than in the previous ten.
Knowing which version added what is the difference between a clean
implementation and a reimplementation of something that already exists.
| Feature | Version |
|---|---|
Path.walk() |
3.12 |
full_match(), Path.from_uri() |
3.13 |
Path.copy, copy_into, move, move_into |
⚠3.14 |
Path.info (cached stat results) |
⚠3.14 |
Path.walk and the pruning idiom
for dirpath, dirnames, filenames in root.walk():
dirnames[:] = [d for d in dirnames if d not in SKIP]
...
That slice assignment is the entire mechanism, and it is worth understanding
rather than memorising. walk yields you the actual list it is about to
recurse into. Mutating it in place — dirnames[:] = ..., not
dirnames = ... — changes which subtrees get visited. Rebinding the name
instead does nothing at all, silently, and your .git and node_modules
directories get walked anyway.
Pruning is not an optimisation you can skip. On a real repository,
node_modules is frequently more files than the rest of the project by an
order of magnitude.
Typing note: walk() yields tuple[Path, list[str], list[str]]. The
first element is a Path; the directory and file names are plain str.
Building a child path is dirpath / name, and treating dirnames entries as
paths is a type error that --strict catches.
💡dirnames = [d for d in dirnames if d not in SKIP] versus
click to reveal
dirnames[:] = [...]. Why does only one prune, and what does that tell you
about walk‘s contract?
Rebinding the local name dirnames points your variable at a new list. The
generator still holds a reference to the original list and iterates that, so
every subtree is visited. Slice assignment mutates the object the generator
holds.
The contract this reveals is unusual and worth naming: walk hands the caller
a mutable window into its own internal state, and the documented way to
influence the traversal is to modify it. That is a side channel, not a
parameter — there is no skip= argument, and there could not easily be one,
because the decision depends on what the walk has already found.
It also means the pruning is invisible to a type checker (both lines
type-check identically) and invisible to a reviewer skimming for = versus
[:]=. When you reimplement walk — as this item’s problem asks you to — the
same property has to hold: yield the list, then iterate it, so the caller’s
edit lands in between.
glob, and the 3.13 change
Path.glob(pattern) and rglob are lazy generators. ** matches any number
of directories.
3.13 silently changed the behaviour of a trailing `**: it now returns files *and* directories, where previously it returned only directories. Code that didfor p in root.glob(““): p.is_dir()and code that assumed the opposite both changed meaning across that boundary, with no deprecation warning. If you have a trailing` anywhere, check it.
full_match() (3.13) matches a pattern against the whole path, including
** semantics — as opposed to match(), which matches from the right and
does not support ** recursion. full_match is what you want for
gitignore-style rules.
Path.info (3.14) and the stat-per-check problem
The naive tree walk is quietly expensive:
for p in root.rglob("*"):
if p.is_file() and p.stat().st_size > threshold: # two stat calls
...
Each is_file(), is_dir(), exists() and stat() is a separate syscall on
the same inode. Over a large tree that is the dominant cost. Path.info
(3.14) caches the stat result on the path object, so repeated questions about
the same path are answered once.
Before 3.14, the equivalent is os.scandir, whose DirEntry objects carry
cached is_file() / is_dir() / stat() from the directory read itself.
Path.walk uses scandir internally, which is one more reason to prefer it
over a hand-rolled recursion.
💡copy, move, copy_into and move_into arrive in 3.14. What
click to reveal
did people use before, and why is the addition more than sugar?
shutil.copy2, shutil.move, and os.replace. The problem was never that
they did not work — it was that a pathlib-based codebase had to keep
converting: shutil.copy2(str(src), str(dst)), or rely on shutil accepting
path-like objects, and then get back a str rather than a Path.
Every one of those conversions is a place where the str type stops carrying
the “this is a path” information, and — more practically — a place where the
result gets re-joined with / or os.path.join and the absolute-path
discard bug can reappear.
The other half is that the methods are defined on the ABC, so alternative
path implementations (zip members, cloud object stores, in-memory
filesystems) can implement them. shutil is hardwired to the local
filesystem.
If you support 3.12 and 3.13, keep using shutil — just pass and return
Path on both sides.