Load Outside Data Without Running Its Code

Your service receives job requests over HTTP. Each request body is a few bytes of JSON, and you need a JobRequest with a label and a number of attempts. The previous article, Bytes, Text and Line Endings, showed how to decode bytes into text. This parser goes one step further, from bytes to a record your code can trust:

import json
from dataclasses import dataclass


@dataclass(frozen=True)
class JobRequest:
    label: str
    attempts: int


def parse_job(payload: bytes) -> JobRequest:
    if len(payload) > 4096:
        raise ValueError("request is over 4096 bytes")
    try:
        value: object = json.loads(payload.decode("utf-8"))
    except RecursionError as exc:
        raise ValueError("request is nested too deeply") from exc
    if not isinstance(value, dict) or set(value) != {"label", "attempts"}:
        raise ValueError("expected exactly two fields, label and attempts")
    label = value["label"]
    attempts = value["attempts"]
    if not isinstance(label, str) or not 1 <= len(label) <= 80:
        raise ValueError("label must be 1 to 80 characters")
    if type(attempts) is not int or not 1 <= attempts <= 5:
        raise ValueError("attempts must be a whole number from 1 to 5")
    return JobRequest(label, attempts)


print(parse_job(b'{"label": "train", "attempts": 2}'))
# JobRequest(label='train', attempts=2)

parse_job refuses a large input before doing any work. It decodes with a format that can build only plain data, and it checks the shape and types before it builds the record.

Never unpickle bytes from outside

pickle turns almost any Python object into bytes and back, so messages and caches need no conversion code. But loading a pickle can call any function the bytes name. This class makes pickle call print while it loads:

import pickle


class Sneaky:
    def __reduce__(self) -> tuple[object, tuple[str]]:
        return (print, ("this ran while loading",))


payload = pickle.dumps(Sneaky())
pickle.loads(payload)  # prints: this ran while loading

Whoever can write those bytes can name os.system instead. Load a pickle only when your own program wrote it and nobody else could change it. The same goes for shelve, which stores pickles, and for marshal.

PyTorch checkpoints hold pickles too. Since PyTorch 2.6, torch.load defaults to weights_only=True, which builds only tensors, plain Python values and the types you allow with torch.serialization.add_safe_globals. Keep that default on.

Check your understanding A Redis cache sits on a private network. Is it safe to unpickle what you read from it? Show answer

Only if everyone who can write to the cache is as trusted as your own code. The network limits who can connect, but other services and anyone with the password can still write. Store JSON and check its shape when you read it back. Then a bad entry raises ValueError instead of running code.

Decode, then check the shape

json.loads builds only dicts, lists, strings, numbers, True, False and None, so it cannot run code. The result is still outside data, so parse_job checks it:

The set(value) check rejects missing and extra fields.
type(attempts) is not int rejects JSON's true. It becomes Python's True, and bool is a subclass of int, so isinstance would accept it.
Bad UTF-8 and bad JSON both raise subclasses of ValueError, so callers need only except ValueError.
A million nested [ characters make json.loads raise RecursionError. The 4096-byte limit stops that here. parse_job still turns a RecursionError into a ValueError, in case someone raises the size limit later.

For YAML, call yaml.safe_load, which builds only plain data, and check the shape the same way. Never pass outside data to yaml.load with yaml.Loader, which can build any Python object.

Limit the work one input can cause

A small input can still use a lot of memory or time. Set limits for what you accept:

Count bytes as they arrive and stop at your limit. A Content-Length header is only what the client claims.
Limit the size after decompression. Compressed data can grow a thousand times or more.
Limit nesting depth and item counts when your parser does not.
Give the whole operation a time limit.

Check archive members before you extract them

A tar or ZIP file stores a path for each file inside it, called a member. A hostile archive can use ../escape.txt to write outside your folder. Check counts and sizes first, then extract with the data filter:

import tarfile

MAX_MEMBERS = 1_000
MAX_TOTAL_BYTES = 100_000_000

with tarfile.open("upload.tar.gz") as archive:
    members = archive.getmembers()
    if len(members) > MAX_MEMBERS:
        raise ValueError("the archive has too many members")
    if sum(member.size for member in members) > MAX_TOTAL_BYTES:
        raise ValueError("the archive is too big once unpacked")
    archive.extractall("unpacked", filter="data")

The data filter refuses device files, and paths or links that point outside the destination. For ../escape.txt it raises tarfile.OutsideDestinationError. Python 3.14 uses this filter by default, but on 3.12 and 3.13 extractall() writes ../escape.txt outside the folder. Always pass filter="data".

When a member fails the filter, the members already written stay on disk. Extract into a new, empty folder, and delete it if extraction fails.

When zipfile extracts, it drops .. and a leading / from member names, but it has no size limit. Add up info.file_size over archive.infolist() before you extract. zipfile stops reading each member at that size, so the total is a real limit.

Going further

Signing a pickle with HMAC proves that nobody changed it after signing. Anyone with the key can still sign a harmful one.
json.loads keeps the last value of a repeated key. To reject duplicates, pass an object_pairs_hook that checks for them.
Read the pickle warning, tar extraction filters, the PyYAML documentation and PyTorch's serialization notes.