Retry, backoff, and retryable vs terminal

A retry loop with no budget, no backoff and no distinction between retryable and terminal failures is the single most common self-inflicted outage in production software. It has a name — the retry storm. A dependency wobbles, every client retries immediately, the retries are themselves load, the dependency goes from wobbling to down, and now the retries never stop because the thing they are waiting for is being kept down by the retries.

Four properties separate a retry loop that helps from one that hurts:

1. A budget. A bounded number of attempts, decided by the caller, so the worst case is knowable.
2. Backoff. Wait longer after each failure. Exponential is the default because it un-synchronises clients within a few rounds.
3. Jitter. Randomise the delay so a thousand clients that failed together do not retry together. This problem deliberately omits jitter, because a test cannot assert on a random sequence — but in production, backoff without jitter merely converts one thundering herd into a slightly later thundering herd. Add random.uniform(0, delay) and mean it.
4. A retryable/terminal split. Retrying a 404 is not resilience, it is four times the latency for the same failure. Only errors that might resolve themselves are worth another attempt.

And when the budget runs out, raise a distinct error chained to the last failure. RetryExhaustedError tells the caller “the policy gave up”; __cause__ tells them what it gave up on. Losing the last failure is losing the only diagnostic information the whole loop produced.

What to build

def fetch_with_retry(
    fetch: Callable[[str], bytes],
    url: str,
    attempts: int,
    sleep: Callable[[float], None],
) -> bytes:

• attempts < 1 raises ValueError before calling fetch even once.
• Attempt n from 1 to attempts: call fetch(url) and return its result.
• On TransientError: if this was the last attempt, raise RetryExhaustedError(f"giving up on {url} after {attempts} attempts") from the caught error. Otherwise call sleep(BASE_DELAY * 2 ** (n - 1)) and go round again.
• Any other exception — TerminalError included — propagates immediately, with no sleep and no further attempts.

Note the ordering: there is no sleep after the final failure. Sleeping and then giving up wastes the caller’s time for nothing. With BASE_DELAY = 0.5 and four attempts the sleeps are 0.5, 1.0, 2.0 — one fewer than the attempts.

The probe

def solve(url: str, script: Sequence[str], attempts: int) -> dict[str, object]:

Build a fetch that consumes script one entry per call — "transient" raises TransientError(f"connection reset on call {n}"), "terminal" raises TerminalError(f"404 on call {n}") where n is the 1-based call number, and anything else returns f"payload for {url}".encode(). If the script runs out, repeat its last entry. Build a sleep that records its argument instead of sleeping.

Return exactly:

Why sleep is a parameter

This is the dependency-injection lesson arriving early, and it is not ceremony. sleep: Callable[[float], None] is why a test for a policy with a sixteen-second worst case runs in microseconds, and why "sleeps" can be asserted exactly instead of approximately. A retry helper that reaches for time.sleep directly is a retry helper whose backoff schedule is untested, which in practice means wrong.

The same trick generalises: a function that takes its clock, its sleeper and its randomness as parameters is a function you can test. One that imports them is one you can only observe.