Why async exists: blocking, threads, and the cost of waiting

This article explains which problem async solves before the rest of the track shows how it works. Two misunderstandings are common:

• expecting async to make CPU-heavy computation faster;
• reading .await as "run this in parallel". Inside one async function, .await waits for that future to finish before the next line runs.

Async is a way to structure concurrency: it lets a program keep many operations in progress while most of them are waiting, using fewer threads than one per operation. It has its own costs, and for many programs ordinary threads are the simpler choice.

Vocabulary

These terms are used throughout the track.

• A blocking call does not return until its result is available. Reading from a std::net::TcpStream with no data waiting blocks the calling thread until data arrives, the connection closes, an error occurs, or a configured timeout expires.
• A future is a value that represents work that may not have finished. Calling an async fn creates a future; it does not run the function body. The body runs only when something polls the future, and each poll returns either Poll::Ready(value) or Poll::Pending ("not finished yet").
• An executor is the code that polls futures. When a future returns Pending, the executor puts it aside and polls something else.
• A waker is a handle the future keeps so it can tell the executor "poll me again"; it is called when the thing it was waiting for has happened.
• A reactor is the code that watches for those events, such as a socket becoming readable or a timer expiring, and calls the matching wakers.
• A runtime is a package of an executor, a reactor and usually async I/O types and timers.

A short example shows that a future does nothing on its own:

async fn greet() {
    println!("hello");
}

fn main() {
    greet(); // compiler warning: unused implementer of `Future` that must be used
    let fut = greet(); // a future value; the body has not run
    drop(fut); // dropped without being polled
}

This program compiles (with the warning shown) and prints nothing, because no code ever polls either future.

What a blocked thread costs

A program that does not use async can handle many connections by giving each connection its own thread and making blocking calls. While a thread is blocked, the operating system does not schedule it, so it uses almost no CPU time. This design is straightforward to write and debug. Its costs grow with the number of threads:

• Stack memory. Every thread has its own stack. Threads created with std::thread::spawn currently get a 2 MiB stack by default on all Tier-1 platforms (the documentation says the default is platform-dependent and may change, and std::thread::Builder::stack_size or the RUST_MIN_STACK environment variable can override it). The main thread's stack size is chosen by the operating system, for example 8 MiB under a common Linux default and 1 MiB by default on Windows. Operating systems usually reserve the stack's address range up front and supply physical memory only for pages the thread actually touches. So 10,000 spawned threads reserve about 20 GiB of address space at the default size, while the memory they really occupy depends on how deep each stack grows.
• Kernel bookkeeping. Each thread is an entity the kernel tracks and schedules. Operating systems also limit how many threads a process or user may create.
• Context switches. Moving a CPU from one thread to another goes through the kernel, saves and restores registers, and can make caches less effective. The cost of each switch depends on the hardware and operating system.

At ten threads none of this is noticeable. At tens of thousands it can dominate memory use or reach system limits. Handling 10,000 simultaneous connections on one server was named the C10K problem around 1999. Common answers use operating-system readiness interfaces (epoll on Linux, kqueue on BSD and macOS, I/O completion ports on Windows) so that a few threads can wait on many connections at once. Async runtimes are built on those interfaces.

Check your understanding A web server holds 10,000 open connections. Each connection needs the CPU about 0.1% of the time and spends the rest waiting for the network. On average, how many connections need the CPU at once? How many threads does a one-thread-per-connection design use? Show answer Hide answer

On average, 10,000 × 0.001 = 10 connections need the CPU at any instant. A handful of threads can therefore keep up with the computation, although bursts can need more.

A one-thread-per-connection design uses 10,000 threads, almost all of them blocked, each with its own stack and kernel scheduling entry.

With async, each waiting connection is instead a future stored in the program's memory. Its size depends on the state it keeps between .await points. The reactor registers the sockets with the operating system's readiness interface, and a few executor threads poll whichever futures have been woken. The sockets are still kernel objects in both designs. What async removes is the need for one thread, and one stack, per waiting connection.

Concurrency is not parallelism

These two words are often used interchangeably, but they mean different things.

Concurrency means having several tasks in progress during the same period and interleaving work on them. One cook making three dishes, switching between them while each one simmers, is concurrent.

Parallelism means executing several things at the same instant on separate hardware, such as different CPU cores. Three cooks each making one dish are working in parallel.

Each can exist without the other. A single-threaded event loop is concurrent but not parallel. A parallel iterator from the rayon crate, which splits one computation across several threads, is parallel without involving any waiting.

Async provides concurrency. Whether futures also run in parallel depends on the executor: a single-threaded executor polls one future at a time, while a multi-threaded runtime can poll different tasks on different threads at the same moment. Within one task, writing a.await; b.await; runs a to completion and then b. To make two futures progress during the same period, combine them, as you will with join in item 16.14, or run them as separate tasks.

The single-threaded executor in item 16.17 shows concurrency without parallelism: several tasks make interleaved progress on one thread, and the recorded order shows the interleaving.

CPU-bound versus I/O-bound

This distinction decides which tool fits.

I/O-bound work spends most of its time waiting: for a socket, a disk, a database or another service. This is where async can help, because a waiting future usually costs less memory than a blocked thread.

CPU-bound work spends its time computing, so there is no waiting to overlap. Async does not split a computation across cores. A task that computes for a long time without reaching an .await that returns Pending keeps its executor thread busy for the whole poll, and other tasks waiting for that thread are delayed until the poll returns. Some multi-threaded runtimes can move queued tasks to other worker threads, but the busy thread is still unavailable. Item 16.22 shows the effect in a small executor. For long computations, threads are usually the better fit: std::thread::scope, or a thread pool sized near the number of CPU cores. When an async program must do such work, it can hand it to a separate thread, use a runtime's dedicated blocking-work threads, or split it into pieces with yields between them.

A rough starting point:

	a few concurrent operations	tens of thousands of concurrent operations
CPU-bound	ordinary sequential code, or a few threads	a thread pool sized near the number of cores
I/O-bound	one thread per operation is usually simpler	async, or another event-driven design

Only one cell of the table points to async. Measurements, existing libraries and team experience can change the choice, but async is one option among several, not a default upgrade.

Async and speed

Async does not make an individual I/O operation complete sooner; the network or disk takes as long either way. It also adds its own work: polling generated state machines, invoking wakers and managing a run queue. A blocking design pays instead for kernel scheduling and per-thread stacks. Which design has lower latency or higher throughput for a given program depends on the workload, the runtime, the operating system and the level of concurrency, so it has to be measured.

What async reliably changes is the cost of holding a waiting operation: a stored future instead of a blocked thread. That matters most when the number of simultaneous waits is large.

What Rust's standard library provides

The standard library contains:

• the async and .await syntax, implemented by the compiler, for functions, blocks and closures;
• the Future and IntoFuture traits, plus small helpers such as ready, pending and poll_fn, in std::future;
• Poll, Context, Waker, Wake, RawWaker and RawWakerVTable in std::task;
• Pin and the pin! macro in std::pin, and the Unpin marker trait in std::marker (it is also in the prelude).

Apart from Wake, which is built on Arc and lives in the alloc crate, these traits and types are defined in core, so code without the standard library (#![no_std], for example on microcontrollers) can use them too.

The standard library does not contain:

• an executor that runs a future to completion;
• a reactor;
• async timers;
• async sockets or files (std::net, std::fs and std::thread::sleep all block);
• a way to make main async: writing async fn main is rejected with error E0752.

So in a new Cargo project with no dependencies, calling an async fn produces a future, and running it requires code that polls it. This track writes that code.

Check your understanding Why does Rust provide async syntax and the Future trait but no runtime to run them? Show answer Hide answer

Because a runtime embodies scheduling policy, and different programs need different policies.

A microcontroller with no heap allocator may want a fixed set of tasks in statically allocated memory, woken by hardware interrupts; the embassy crate takes this approach. A network server may want several worker threads that take work from each other's queues and an operating-system readiness interface. Many GUI toolkits require user-interface updates to happen on one particular thread. A test may want a deterministic single-threaded executor with simulated time, which is what you build in item 16.19.

One built-in runtime would suit some of these and impose costs on the others. Rust has precedent here: before 1.0, its standard library included an optional green-threading runtime, and RFC 230 removed it in 2014, partly because supporting green and native threads behind one I/O interface added overhead and complexity even for programs that used only native threads.

Instead, the language standardizes the interface: one Future trait and one Waker contract. Futures written against that interface can be combined and polled by any executor. The policy lives in libraries. The cost is that a program has to choose a runtime, and libraries whose I/O types depend on one runtime's reactor may not work under another.

Because the interface is public, you can also build a small executor and reactor yourself using only the standard library, which is what this track does.

What the track covers

The track's exercises use no external crates: no tokio, no smol. Every exercise that runs a future does so with an executor or driver written in ordinary Rust, either supplied in the exercise or written by you, so you can read exactly how each future gets polled.

In order, you will write a future by hand, a block_on function that polls one future to completion, wakers built with the Wake trait and then with RawWaker, a hand-compiled state machine, join and select combinators, a single-threaded executor, spawning with a JoinHandle, and a reactor whose timers run on logical time. Later items cover fairness, cancellation and blocking inside async code, and an article compares these pieces with how tokio and smol are built.