Skip to content
← All Python articles

Python explainer

Retry Failed Calls with Backoff

About 4 min to read. Allow extra time to work through examples.

Retry a failing call only when repeating it is safe, wait longer after each failure, cap the attempts and the time, and test the waits without sleeping.

This article uses Python. You can make it your practice mode to browse related content across the site.

Your app calls a payment service. Now and then a call fails with a timeout, and the same call works a second later. A retry loop tries again, and waits a little longer after each failure:

import time
from collections.abc import Callable


class TransientError(Exception):
    '''A failure that may go away if you try again.'''


def with_retries(call: Callable[[], bytes], attempts: int = 4) -> bytes:
    for attempt in range(1, attempts + 1):
        try:
            return call()
        except TransientError:
            if attempt == attempts:
                raise
            time.sleep(0.5 * 2 ** (attempt - 1))
    raise AssertionError("unreachable")


assert with_retries(lambda: b"paid") == b"paid"

Backoff means waiting longer after each failure. When the wait doubles each time, as with 0.5, 1 and 2 seconds here, it is exponential backoff. The loop retries only TransientError. Any other exception escapes at once, and after the last attempt the final TransientError escapes too.

Retry only what is safe to repeat

A timeout does not tell you whether the service did the work. The charge may have gone through, and only the response got lost. If you send it again, the customer pays twice.

An operation is idempotent if doing it twice has the same effect as doing it once. Reading a record is idempotent, and so is setting a field to a fixed value. Charging a card is not, unless the service gives you a way to make it so. Many payment services accept an idempotency key, a unique string that you send with every attempt at one charge. The service then charges once, however many attempts arrive. Use a key only when the service documents it.

Retry the failures that the service documents as temporary, such as a rate limit or a brief server error. When the service rejects your input, fix the request instead.

Count the first call as an attempt

The attempt limit includes the first call, so four attempts allow three retries. Check that the limit is at least 1 before the first call.

With a base delay of 0.5 seconds that doubles after each failure, four attempts wait at most three times:

attempt 1 fails, wait 0.5
attempt 2 fails, wait 1.0
attempt 3 fails, wait 2.0
attempt 4 fails, stop

That is 3.5 seconds of waiting. A fifth attempt adds a 4-second wait, for 7.5 seconds in all. These totals leave out the time the calls take. A call that hangs can block forever. Give each call a timeout, set a deadline for the whole operation, and check the time left before each wait.

Add jitter so clients spread out

When a server fails for a moment, all its clients fail together. With the same backoff schedule, they also retry together and hit the server at the same moments. Jitter adds randomness to each wait to spread the clients out. One scheme, called full jitter, picks each wait at random between zero and the backoff value.

If the service says how long to wait, for example in a Retry-After header, wait that long. If that is past your deadline, stop instead.

Test the waits without sleeping

Real sleeps make a test slow. Pass the sleep function in as a parameter, so a test can record the waits instead:

from collections.abc import Callable


def wait_between_attempts(sleep: Callable[[float], None]) -> None:
    for delay in (0.5, 1.0, 2.0):
        sleep(delay)


waits: list[float] = []
wait_between_attempts(waits.append)
assert waits == [0.5, 1.0, 2.0]
assert sum(waits) == 3.5

In production you pass time.sleep. A clock or a random number generator can come in the same way. Test a quick success, a success after a few failures, running out of attempts, and an error that must escape at once. Check that the loop never waits after a success or after the last attempt.

Retry in one layer only

Check your understanding How many requests can reach the service if your code and the client library it uses each make up to three attempts? Show answer

Up to nine, because each of your three attempts can become three attempts inside the library.

Let one layer own the retries, and read the library's retry settings before you add a loop of your own.

Tell the caller why you stopped

In the next exercise, Retry a Fetch with Exponential Backoff, the helper raises RetryExhaustedError from the last TransientError when the attempts run out. The new error tells the caller why the helper stopped, and its __cause__ keeps the last failure for the traceback.

Another choice is to re-raise the last error itself and add the attempt count with add_note(). Callers then catch the same error type as before. Pick one, and document it.

Going further

Each client's limit caps only its own retries. When many clients retry at once, the total can overload a service, so large systems add a shared retry budget or a rate limit. To test jitter, pass in the random source too, and check that each wait stays between zero and the backoff value.

step 13 of 20

Error Handling with Exceptions, Retries and Warnings

Up next Retry a Fetch with Exponential Backoff