Your job calls a partner's API once for each of 10,000 user IDs. The partner allows 10 open connections and 100 requests per second, and your worker has a fixed amount of memory. Those are three different limits, and each one needs its own tool:
| Limit | What it caps | A common tool |
|---|---|---|
| Concurrency limit | Calls running at the same time | asyncio.Semaphore, or the pool size |
| Rate limit | Calls started each second | A token bucket |
| Backpressure | Work waiting in memory | A bounded queue |
The bounded-map exercise after this article practises the first and the last together.
A semaphore caps the calls running at once
A semaphore holds a fixed number of permits. async with sem: takes one,
waits if none are left, and gives it back when the block ends.
import asyncio
sem = asyncio.Semaphore(10)
async def fetch_user(user_id: int) -> int:
async with sem: # at most 10 calls inside this block at once
await asyncio.sleep(1) # stands in for the API call
return user_id
async def main() -> None:
async with asyncio.TaskGroup() as tg:
for user_id in range(100):
tg.create_task(fetch_user(user_id))
asyncio.run(main()) # takes about 10 seconds, 10 calls at a time
A semaphore caps the calls running at once, and the rate then depends on how long each call takes. With one-second calls, 10 permits give about 10 calls per second. With one-millisecond calls, the same 10 permits allow about 10,000 per second.
Hold the permit for as long as the call uses the connection. If the call returns a streaming response, keep the permit until you have read it all.
A rate limit needs a clock
A token bucket is the usual rate limiter. The bucket holds up to N tokens and refills at a steady rate. Each call takes a token before it starts, and waits when the bucket is empty. A full bucket allows a burst of N calls.
Providers define "100 per second" in different ways, such as a token bucket or a fixed or sliding one-second window. Read the provider's definition and match it.
A rate limit on its own leaves the number of running calls unbounded. If the partner slows to 5 seconds per call, 100 starts per second means 500 calls running at once. Use a semaphore as well.
Check your understanding Should a call take its rate-limit token before or after its semaphore permit? Show answer Hide answer
It depends on what the provider counts. If it counts request starts, take the permit first and the token last, right before you send. If you take tokens first, calls can pile up holding tokens while they wait for permits. When permits free up, those calls all start together, and that burst can exceed the provider's limit.
The cost of taking the permit first is that a permit sits idle while its call
waits for a token. Take the permit with async with, so a cancelled call
gives it back.
Bound the queued work too
The semaphore example creates all 100 tasks up front. With 10,000 IDs, you
would hold 10,000 tasks and their arguments in memory, most of them waiting
for a permit. To bound memory, stop reading input while enough work is
waiting. Run a fixed number of workers that read from a bounded
asyncio.Queue, as in the queue article.
Thread pools have the same problem. executor.map reads the whole input and
submits every item as soon as you call it. The pool size limits running work,
and the queue of submitted work can still grow without limit. On Python 3.14,
pass buffersize to keep at most that many items submitted ahead of the
results you have read. On older versions, submit in batches, or keep a
rolling window of futures and submit a new item each time one finishes.
Read more in the Executor.map documentation.