Walk a Directory Tree and Skip Folders

You want every .py file in a project, but not the ones under .venv or node_modules. Those folders can hold thousands of files. If you filter them out of the results afterwards, the program still reads every one of them first. Path.walk(), new in Python 3.12, lets you skip a folder before the walk goes into it.

import tempfile
from pathlib import Path


def python_files(root: Path, skip: set[str]) -> list[Path]:
    found: list[Path] = []
    for folder, dirnames, filenames in root.walk():
        dirnames[:] = [name for name in dirnames if name not in skip]
        for name in filenames:
            if name.endswith(".py"):
                found.append(folder / name)
    return sorted(found)


with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    for rel in ["app/main.py", "app/util.py", ".venv/lib/site.py", "README.md"]:
        (root / rel).parent.mkdir(parents=True, exist_ok=True)
        (root / rel).write_text("", encoding="utf-8")
    for path in python_files(root, {".venv"}):
        print(path.relative_to(root))
# app/main.py
# app/util.py

How the walk works

root.walk() returns a generator. For each folder it visits, it yields three things. The first is the folder as a Path. The second is a list of the names of its subfolders, and the third is a list of the names of its files. The names are plain strings, so join one onto the folder with / to get its path.

By default the walk is top-down. It yields a folder before it visits that folder's subfolders. As you saw in "Generators and One-Shot Iterators", the generator then waits at its yield until you ask for the next item. When it resumes, it visits the names still in dirnames. So if your loop body changes that list, it changes where the walk goes next.

Change the list in place

dirnames[:] = [...] replaces the contents of the list that the walk holds. A common mistake is dirnames = [...]. That points your local variable at a new list, the walk keeps using the old one, and nothing gets skipped. You can also sort or reverse dirnames in place to change the order of the visits.

With walk(top_down=False), the walk yields each folder after its subfolders. By then it is too late to skip anything.

Check your understanding Why remove folders from dirnames instead of filtering the final list? Show answer

Removing a folder from dirnames means the walk never opens it. It never reads the folder's files, and it never hits an error inside it. Filtering the final list can give the same result, but only after the program has read every skipped folder. In the exercise you write the walk yourself, over a folder tree stored in a dictionary, so you control this step directly.

A real disk needs a few decisions

Order. The operating system lists a folder's entries in no fixed order. Sort them when the order matters.
Errors. By default, walk() silently skips a folder it cannot list, for example because of its permissions. Pass a function as on_error to log or raise those errors when a missing folder would make the result wrong.
Symlinks. A symlink is a file that points to another path. By default, walk() does not follow symlinks to folders, and it lists them among the file names. With follow_symlinks=True it follows them. Then a link that points back up the tree makes the walk visit the same folders again and again.

Going further

root.glob("**/*.py") and root.rglob("*.py") find paths that match a pattern. Use them when a pattern describes what you want. They cannot skip a folder partway through, and their order is not fixed. From Python 3.13, a pattern that ends in ** matches files as well as folders.
PurePath.full_match(), new in 3.13, matches a whole path against a pattern, with ** matching any number of folders.
Python 3.14 added Path.copy(), copy_into(), move() and move_into(). On older versions, pass Path objects straight to the functions in shutil.
See the Path.walk() documentation.