This item is a cross-reference. The problem lives at 7.6 — Bounded concurrency and backpressure, in the laziness and pipelines track, because the mechanism is a property of the pipeline rather than of asyncio.
Two things are worth adding here, from the concurrency side.
A concurrency limiter is not a rate limiter
They are constantly conflated, and they solve different problems.
A concurrency limiter bounds how many operations are in flight at once. It needs no clock:
sem = asyncio.Semaphore(10)
async def fetch(key: str) -> bytes:
async with sem:
return await client.get(key)Ten at a time, forever. If each takes a millisecond you are doing 10,000 per second; if each takes a second you are doing ten. The rate is an emergent property, not a controlled one.
A rate limiter bounds operations per unit time, and therefore needs a clock: a token bucket refilled on a schedule, or a sliding window over timestamps. It says nothing about how many are in flight — a limiter of 100/s against a service that has become slow will happily have 1,000 requests outstanding.
Which one you need follows from what the downstream constraint actually is:
| Downstream limit | Tool |
|---|---|
| Connection pool size, worker count, memory per in-flight item | concurrency limiter |
| “1000 requests per minute” in someone’s API terms | rate limiter |
| Both (the common case) | both, composed |
Using a semaphore where the contract is a rate gets you throttled by the provider. Using a rate limiter where the constraint is memory gets you an OOM the first time the downstream slows down.
💡A downstream API allows 100 requests per second. You set Semaphore(100). What happens?
click to reveal
It works until the API gets slow, and then it fails in both directions at once.
When latency is 10 ms, a semaphore of 100 admits roughly 10,000 requests per second — a hundred times over the limit, so you get 429s. When latency rises to 2 seconds, the same semaphore admits 50 per second, so you are now under-using your allowance while your own queue backs up.
The semaphore is holding concurrency constant, and the thing you were asked to hold constant was rate. The two are related by latency, which is exactly the variable you do not control.
The usual production answer is both: a token bucket for the contractual rate, and a semaphore for your own memory and connection ceiling. They compose cleanly — acquire the token, then acquire the slot — and the semaphore is what stops a latency spike from turning into unbounded outstanding work.
The sync-side mirror: Executor.map(buffersize=)
The same lesson exists on the threads side and arrived much later.
Executor.map consumes its entire input iterable immediately, submitting every
item as a task. Feed it a lazy reader over a large file and you have
materialised the whole thing, with no backpressure at all — which is the exact
failure that unbounded asyncio.gather over a big list produces.
Python 3.14 added buffersize= to Executor.map for precisely this: keep at
most N items submitted, and pull from the source only as results are consumed.
On 3.12 and 3.13 the workaround is to chunk the input yourself with
itertools.batched and map over one batch at a time.
That both APIs eventually needed the same bound is the point worth taking away. Unbounded fan-out is the same bug in both models, and it is the most common way an async rewrite makes throughput worse rather than better: the rewrite removes the natural backpressure that a thread pool’s fixed size was providing for free.