Retry, backoff, and the difference between retryable and terminal

The retry loop is the most reliably self-inflicted outage in production software. Not because retrying is wrong — it is usually right — but because the naive version has a positive feedback loop built into it, and positive feedback loops in distributed systems have a name: retry storm.

The mechanism is simple enough to draw. A dependency slows down. Every client times out. Every client immediately retries. The retries are themselves load, so the dependency slows down further. Now it is failing, so every client retries three times instead of once, tripling the offered load at exactly the moment the service can least afford it. The dependency does not recover, because the thing keeping it down is the traffic trying to determine whether it is up.

You cannot fix that with a better exception hierarchy. You fix it with policy.

The four properties

1. A budget. A bounded attempt count, chosen by the caller, so the worst case is knowable. attempts: int in the signature, validated before the first call — attempts=0 should be a ValueError, not a silent no-op that returns None and confuses somebody at 3am.

The budget is a latency contract as much as a reliability one. Five attempts with exponential backoff from 0.5s is a worst case of about eight seconds plus five request timeouts. If the caller is an HTTP handler with a 5s SLA, your retry policy has just guaranteed a timeout instead of preventing one.

2. Backoff. Wait longer after each failure. Exponential is the default — base * 2 ** (n - 1) — because it un-synchronises clients within a few rounds and because it costs nothing to implement. Linear backoff is defensible when the failure is known to be short. No backoff is defensible essentially never.

3. Jitter. Randomise the delay:

time.sleep(random.uniform(0, base * 2 ** (n - 1)))

Without it, a thousand clients that failed together at $t$ retry together at $t + 0.5$, and again at $t + 1.5$, and again at $t + 3.5$. Exponential backoff with no jitter converts one thundering herd into a series of slightly later, equally synchronised thundering herds. “Full jitter” — a uniform draw over $[0, delay]$ — is the variant that performs best in practice and is the one to reach for by default.

4. A retryable/terminal split. This is the property people skip and the one that does the most damage. Retrying a 404 is not resilience; it is four times the latency for exactly the same failure. Retrying a 400 because the payload was malformed is worse: it is four times the latency and four times the load for a request that was never going to work.

A workable default for HTTP:

retryable	terminal
connection errors, read timeouts	400, 401, 403, 404, 422
429 (honour Retry-After)	any error whose cause is your own payload
500, 502, 503, 504	anything you cannot make idempotent

💡Why is idempotency in that table at all? It is not an error class. click to reveal

Because a retry is a second execution, and whether a second execution is safe is a property of the operation, not of the failure.

Consider a POST /charges that times out. You do not know whether the server never saw the request, saw it and crashed before charging, or charged the card and crashed before replying. Retrying is a coin flip on double-charging a customer.

This is why the correct fix is almost never “retry harder”. It is an idempotency key: the client generates a unique id per logical operation, sends it with every attempt, and the server deduplicates. Now the retry is safe by construction and the policy question becomes a pure latency question.

The rule that falls out: only retry operations that are idempotent, or that you have made idempotent. GET and PUT are usually fine. POST is not, until you have done the work. A retry decorator applied indiscriminately across a service is a correctness bug hiding inside a reliability feature.

Injecting sleep is not ceremony

def fetch_with_retry(
    fetch: Callable[[str], bytes],
    url: str,
    attempts: int,
    sleep: Callable[[float], None],
) -> bytes:

That fourth parameter is the difference between a retry policy you have tested and one you have hoped about. With time.sleep hard-coded, a test for a policy whose worst case is sixteen seconds either takes sixteen seconds or gets mocked with a patch that couples the test to an import path. With sleep injected, the test runs in microseconds and can assert the exact backoff sequence — [0.5, 1.0, 2.0] — rather than “it slept for a while”.

Asserting the sequence catches the off-by-one that every hand-written retry loop has at least once: sleeping after the final failure. There is nothing to wait for; you are about to give up. Four attempts means three sleeps.

The same trick generalises to every ambient dependency: pass the clock, pass the randomness, pass the sleeper. A function that takes them is testable; one that imports them is merely observable.

💡Where should the retry live — in the client library, in the caller, or in the service mesh? click to reveal

One of them, and you have to decide which. The pathology is retries at multiple layers, because they multiply rather than add: three retries in the SDK inside three retries in the caller is nine requests for one logical operation, and neither layer knows.

The usual answer is as low as possible while still knowing whether it is safe. The client library knows what a 429 means and how to read Retry-After; it does not know whether your POST is idempotent. So: the library exposes the classification (a TransientError type, a retry_after attribute) and the caller owns the policy.

If you have a mesh doing retries, turn the application-level ones off — or at minimum make the total attempt count across layers an explicit, documented number rather than an emergent one.

And whatever you choose, add a retry budget — a cap on the ratio of retries to requests across the whole client, not per call site. That is the mechanism that actually prevents a storm, because per-call limits still allow every caller to triple its load simultaneously.

When the budget runs out

Raise a distinct error, chained to the last failure:

raise RetryExhaustedError(f"giving up on {url} after {attempts} attempts") from exc

The class says the policy gave up. The __cause__ says what it gave up on. Losing either one loses half the diagnosis: a bare re-raise tells the caller nothing about how hard you tried, and an unchained RetryExhaustedError tells them nothing about what kept failing.

Add a note on the way out if you have anything cheap and useful — exc.add_note(f"attempts={n} elapsed={t:.1f}s") — because that information survives even if a layer above wraps your error again.

The checklist

• Bounded attempts, validated, with the latency budget written down next to it.
• Exponential backoff.
• Full jitter. Always.
• Retryable versus terminal, decided by class, not by string matching.
• Only retry what is idempotent, or make it idempotent with a key.
• One retry layer, not three.
• Inject the sleeper so the schedule is tested rather than assumed.
• On exhaustion, a distinct error chained to the last failure.