Choosing a Concurrency Model: A Decision Procedure

Most concurrency rewrites are motivated by a misdiagnosis. Before choosing a model, decide which of four situations you are actually in — the answer is usually determined by facts you can measure in an afternoon.

The four cases

1. Blocking I/O — threads or asyncio

Sockets, files, subprocesses, database drivers. The GIL is released for the whole wait, so threads genuinely wait in parallel. Both models work.

Choose asyncio when the entire call graph is already async-native (you are in FastAPI, aiohttp, asyncpg territory) or when you need tens of thousands of concurrent operations, where a thread each would cost gigabytes of stack. Choose threads when the libraries you must use are blocking and the concurrency you need is in the dozens or low hundreds. A ThreadPoolExecutor around a blocking driver is not a compromise; it is the correct answer to that shape of problem.

2. CPU-bound pure Python — processes, today

A tight loop over Python objects holds the GIL. Threads will not help; they will make it marginally worse.

Processes are the answer with today’s dependency sets. Subinterpreters (concurrent.interpreters, 3.14) are attractive when payloads are small and workers are long-lived, because they avoid the fork/spawn cost while still avoiding the GIL — but every extension module in your stack must be subinterpreter-safe. Free-threading is supported (not experimental) from 3.14 and is the eventual answer, gated on your dependencies shipping free-threaded wheels. Track 10 covers all three properly.

3. CPU-bound inside a C library that releases the GIL — threads already work

This is the one people get wrong in both directions, and it is worth internalising.

NumPy’s array kernels, Pillow’s codecs, zlib, hashlib, bz2, lzma, compiled ML kernels — these release the GIL around the expensive part. Which means:

• Reaching for multiprocessing to parallelise a NumPy pipeline pays for pickling large arrays and duplicating memory, to buy something threads were already giving you.
• Concluding “Python can’t do CPU parallelism” and rewriting in another language is, for this workload, simply false.

The mistake in the other direction is assuming all of a library releases the GIL. NumPy releases it for large-array elementwise work and BLAS calls; it does not for scalar operations, object arrays, or the Python-level glue between kernels. Measure before you conclude.

4. You need a hard cancellation story — asyncio

This is the strongest single argument for asyncio in a service, and it is structural rather than a matter of taste: there is no safe way to cancel a running thread in Python. No API, and not by oversight — asynchronously injecting an exception into arbitrary code would break every invariant any lock, open file or half-updated structure was relying on.

So a thread that has stopped making progress can be asked to stop (via a flag it polls) and otherwise cannot be stopped at all. Your shutdown path is “set the event, join with a timeout, log and abandon”. If your orchestrator SIGKILLs after 30 seconds and the thread is in a 60-second socket read, you lose the work.

A Task can be cancelled at any suspension point, with finally blocks running. That is what makes a bounded drain possible, and it is why services with a strict shutdown SLA end up async.

💡A service is at 100% CPU on one core with seven idle. py-spy shows most samples inside numpy.dot. What should change? click to reveal

Almost certainly nothing about the concurrency model — and quite possibly the BLAS configuration.

numpy.dot releases the GIL and dispatches to BLAS, which is already multi-threaded in most builds. One busy core out of eight suggests BLAS has been pinned to a single thread: OMP_NUM_THREADS=1 or OPENBLAS_NUM_THREADS=1 in the environment, which container images and some ML frameworks set deliberately to avoid oversubscription. Check that first; it is a one-line fix that no amount of Python-level restructuring will substitute for.

If BLAS is genuinely single-threaded on purpose (because you are running eight worker processes and want one core each), then the system is behaving as designed and the answer is more workers, not more threads.

And if you do want to parallelise at the Python level over independent matrices, threads are the right tool precisely because dot releases the GIL — ThreadPoolExecutor, not ProcessPoolExecutor, and no pickling of large arrays.

The general lesson: profile before choosing a model, and when the hot frame is in a C extension, find out whether it releases the GIL before assuming Python is the constraint.

The mixing rules

Real systems combine models. Four rules keep that safe:

1. One event loop per thread. Not one per process — a worker thread may run its own loop — but never two on one thread.
2. async → blocking: asyncio.to_thread(fn, *args), or loop.run_in_executor(pool, fn) when you want to control which pool. Never call a blocking function directly on the loop thread.
3. blocking → async: loop.call_soon_threadsafe(cb) to schedule a callback, asyncio.run_coroutine_threadsafe(coro, loop) to run a coroutine and get a concurrent.futures.Future back. These two are the only thread-safe entry points; everything else in asyncio is documented as not thread-safe, and debug mode will raise if you get it wrong.
4. Never asyncio.run() on a thread that already has a running loop. That is the check item 9.6 builds.

The procedure, compressed

1. Measure CPU time against wall time. Low ratio → you are waiting; the GIL is not your problem.
2. If you are waiting: threads if the ecosystem is blocking, asyncio if it is async-native or the concurrency is huge.
3. If you are computing: find out whether the hot frame is pure Python or a C extension that releases the GIL. Threads already work for the second.
4. If it is pure Python: processes today, subinterpreters or free-threading as your dependencies allow.
5. Override all of the above with asyncio if a hard, bounded shutdown is a requirement — because that is the one thing threads cannot give you at any price.