SIMD (single instruction, multiple data) means one CPU instruction operating on several values at once, held side by side in a wide register. Each value position in the register is called a lane: a 128-bit register holds four f32 lanes, and a 256-bit register holds eight.
Rust gives you three ways to get SIMD code. They differ in whether they work on stable Rust, whether the same source works on every CPU architecture, and whether they need unsafe. This article describes each one as of Rust 1.98 and ends with the order in which to try them.
Two more terms first. A target is the platform a program is compiled for, such as x86_64-unknown-linux-gnu or aarch64-apple-darwin. A target feature is an optional instruction set extension that some CPUs of that architecture have and others do not, such as AVX or AVX2 on x86-64.
Tier 1: auto-vectorisation
Stable, portable and safe. You write ordinary loops.
When optimisation is on (for example cargo build --release), LLVM, the code generator rustc uses, may turn an ordinary loop into vector instructions if it can show that processing several iterations at once gives the same result. There are no annotations and no unsafe, and the same source can be vectorised for any target. The default debug profile (opt-level = 0) does not do this.
Which vector instructions the compiler may use is fixed at compile time. By default it uses only the features the target guarantees. On 64-bit ARM that includes the 128-bit NEON instructions. On x86_64-unknown-linux-gnu it is SSE and SSE2, which are also 128 bits wide; some other x86-64 targets assume a little more. The 256-bit AVX2 or 512-bit AVX-512 instructions are used only if you enable those features when building (see the end of tier 2).
Before vectorising a loop, LLVM has to establish that:
- the iterations do not depend on each other, apart from patterns it knows how to handle, such as a reduction (a loop that combines all the elements into one value, like a sum);
- the memory the loop writes does not overlap the memory it reads in a way that would change the result;
- for a reduction, regrouping the operations into several partial results does not change the answer.
The last point explains a common surprise. Integer addition in a release build wraps on overflow, and wrapping addition gives the same answer in any grouping, so LLVM can split an integer sum into several partial sums, one per lane, and add the lanes together at the end. Floating-point addition rounds after every step, so (a + b) + c can differ from a + (b + c). Rust does not allow the compiler to regroup float operations, so this dot product must perform its additions one after another, in order:
fn dot(a: &[f32], b: &[f32]) -> f32 {
let mut sum = 0.0;
for (x, y) in a.iter().zip(b) {
sum += x * y;
}
sum
}
Each addition has to wait for the previous one, so the additions cannot be spread across the lanes of a vector register. Depending on how the loop is written, LLVM may still compute several multiplications at once and then add the products one at a time, but the chain of additions stays serial. You can choose a different grouping yourself by keeping several independent partial sums and combining them at the end:
fn dot8(a: &[f32], b: &[f32]) -> f32 {
assert_eq!(a.len(), b.len());
// Split each slice into whole chunks of eight plus the leftover elements.
let (a_chunks, a_tail) = a.as_chunks::<8>();
let (b_chunks, b_tail) = b.as_chunks::<8>();
let mut acc = [0.0f32; 8];
for (ca, cb) in a_chunks.iter().zip(b_chunks) {
for ((s, x), y) in acc.iter_mut().zip(ca).zip(cb) {
*s += x * y;
}
}
let tail: f32 = a_tail.iter().zip(b_tail).map(|(x, y)| x * y).sum();
acc.iter().sum::<f32>() + tail
}
as_chunks::<8>() (stable since Rust 1.88) returns a slice of [f32; 8] arrays covering as much of the input as possible, plus the fewer than eight elements left over at the end. Each of the eight entries of acc is its own chain of additions, independent of the other seven, so the eight chains can run side by side in vector registers. Two consequences:
dot8adds in a different order fromdot, so the two can return slightly different values for the same input.dot8is still deterministic: the same input always gives the same output.dot8panics if the lengths differ, whiledotsilently stops at the end of the shorter slice. Choose the behaviour you want and document it.
How much faster dot8 is depends on the CPU, the data size, and whether the data is already in cache. Measure it on the hardware you care about rather than relying on a number from someone else's machine.
Check your understanding You have a loop that you expected to vectorise and it did not. What are the usual causes, and how do you find out which one you have? Show answer Hide answer
Common causes:
A floating-point reduction. sum += a[i] * b[i] cannot be regrouped, as described above. LLVM may vectorise the multiplications and still add the products one by one in order, so the loop can be reported as vectorised and gain little. Several independent accumulators fix it.
Possible overlap between memory written and memory read. Rust tells LLVM that a &mut reference does not overlap any other reference that is in use at the same time, so a loop that reads from a &[f32] and writes to a &mut [f32] can be vectorised without overlap checks. Raw pointers carry no such promise, so a loop that writes through one *mut f32 and reads through another may get runtime overlap checks or stay scalar. When you need two non-overlapping mutable parts of one slice, split_at_mut gives you two &mut slices that the type system knows are separate.
An exit from the middle of the loop. break, return or ? inside the body means the loop may stop partway, at a point that depends on the data. An index that might be out of bounds is also an exit, because it panics. LLVM removes a bounds check when it can prove the index is in range, so iterating with zip, as_chunks or chunks_exact, or putting one assert! that relates the lengths before the loop, often removes the per-iteration checks.
An operation with no vector form. A call to a function that was not inlined, or an operation the target has no vector instruction for, keeps that part of the loop scalar.
A debug build. At opt-level = 0 nothing is vectorised, and integer overflow checks are on as well.
How to find out: look at the generated assembly, either with cargo rustc --release -- --emit asm (the .s file appears under target/release/deps/) or with the third-party cargo-show-asm tool, which provides cargo asm. On x86-64, scalar and vector floating-point code both use xmm registers, so look at the instruction names: addps and mulps (with a v prefix when AVX encoding is used, for example vaddps) work on packed lanes, while addss and mulss work on one value. A 256-bit ymm or 512-bit zmm register always means vector code. On 64-bit ARM, vector operands look like v0.4s (four 32-bit lanes), while scalar ones look like s0.
LLVM can also report its decisions. rustc -C opt-level=3 -C remark=loop-vectorize prints a note for each loop the loop vectoriser handled, either "vectorized loop" with the vector width or a short reason it was not, such as "the cost-model indicates that vectorization is not beneficial". Without debug information the notes show <unknown file>:0:0; with -C debuginfo=1 they show a file and line, which for iterator-based loops is often a line inside the standard library code that was inlined into your function. A "vectorized" note also does not tell you whether the additions were the part that was vectorised, so the assembly remains the reliable check.
Tier 2: core::arch intrinsics
Stable, but specific to one architecture, and partly unsafe.
core::arch (also available as std::arch) exposes individual CPU instructions as functions called intrinsics, using the names from the CPU vendor's documentation, such as _mm256_add_ps on x86-64. The x86 and x86-64 intrinsics became stable in Rust 1.27, and the 64-bit ARM NEON intrinsics in Rust 1.59. Intrinsics for some other architectures are still unstable.
This function sums a slice with 256-bit AVX instructions when the CPU supports them, and with ordinary code otherwise:
#[cfg(target_arch = "x86_64")]
use std::arch::x86_64::{_mm256_add_ps, _mm256_loadu_ps, _mm256_setzero_ps, _mm256_storeu_ps};
fn sum_scalar(v: &[f32]) -> f32 {
v.iter().sum()
}
#[cfg(target_arch = "x86_64")]
#[target_feature(enable = "avx")]
fn sum_avx(v: &[f32]) -> f32 {
let (chunks, tail) = v.as_chunks::<8>();
let mut acc = _mm256_setzero_ps();
for chunk in chunks {
// SAFETY: `chunk` is a `[f32; 8]`, and this load reads 8 f32 values.
let x = unsafe { _mm256_loadu_ps(chunk.as_ptr()) };
acc = _mm256_add_ps(acc, x);
}
let mut lanes = [0.0f32; 8];
// SAFETY: `lanes` has room for the 8 f32 values this store writes.
unsafe { _mm256_storeu_ps(lanes.as_mut_ptr(), acc) };
lanes.iter().sum::<f32>() + sum_scalar(tail)
}
pub fn sum(v: &[f32]) -> f32 {
#[cfg(target_arch = "x86_64")]
if is_x86_feature_detected!("avx") {
// SAFETY: the check above confirmed that this CPU supports AVX.
return unsafe { sum_avx(v) };
}
sum_scalar(v)
}
Everything in it is stable. What each part does:
#[cfg(target_arch = "x86_64")]removes the AVX code entirely when compiling for another architecture, where those intrinsics do not exist. On 64-bit ARM,sumis justsum_scalar.#[target_feature(enable = "avx")]compilessum_avxas if the CPU had AVX. Running it on a CPU without AVX is undefined behaviour. In practice the program often crashes with an illegal-instruction signal, but Rust makes no promise about what happens, so you cannot rely on catching it.- Since Rust 1.86 a
#[target_feature]function can be a safefn. Calling it withoutunsafeis allowed only from another function whose own#[target_feature]attribute enables the same features; enabling the feature for the whole build with-C target-featuredoes not count. Every other call needsunsafe, because the caller has to guarantee that the CPU really has the feature. Hereis_x86_feature_detected!("avx")checks the CPU at run time, and theSAFETYcomment records that. - Since Rust 1.87, most intrinsics that take no pointer arguments, such as
_mm256_setzero_psand_mm256_add_ps, are safe to call inside a function whose own#[target_feature]attribute enables the feature they need. Enabling the feature for the whole program does not count: outside such a function, even an intrinsic for a feature the target always has (SSE2 on x86-64, NEON on 64-bit ARM) still needs anunsafeblock. Intrinsics that read or write through a raw pointer, such as_mm256_loadu_psand_mm256_storeu_ps, are alwaysunsafe, because the compiler cannot check the pointer. is_x86_feature_detected!is defined instd, notcore, sono_stdcode (code that does not link the standard library, such as firmware) cannot use it on stable Rust. ARM has the equivalentstd::arch::is_aarch64_feature_detected!.- The scalar fallback is still required, so the intrinsic version is extra code for each architecture you support, and it needs its own tests. Note also that
sum_avxadds in a different order fromsum_scalar, so the two paths can return slightly different results for the same input. Tests should compare them with a tolerance, and the function's documentation should say that the result can vary by machine.
A #[target_feature] function cannot also be marked #[inline(always)] on stable Rust; the compiler rejects the combination. Plain #[inline] is accepted, but the compiler generally inlines such a function only into callers that enable the same features.
There is also a compile-time route. Building with -C target-feature=+avx2, or with -C target-cpu=native (which enables every feature of the CPU doing the build), lets the compiler use those instructions anywhere in the program, so ordinary safe code, including auto-vectorised loops, benefits without any unsafe. It also makes cfg(target_feature = "avx2") true, so code can choose a path at compile time instead of checking at run time. This is a good choice when you control the machines the program will run on. A binary built this way and then run on a CPU without those features has undefined behaviour, typically an illegal-instruction crash, so it is a poor choice for binaries you distribute to other people.
When you need to choose between several cfg conditions, the cfg_select! macro, stable since Rust 1.95, picks the first branch whose condition holds and can be used where an item, statement or expression is expected. It covers the same need as the third-party cfg-if crate.
Tier 3: std::simd (portable_simd)
Nightly only as of Rust 1.98.
#![feature(portable_simd)]
use std::simd::f32x8;
On a stable compiler the first line is rejected with error E0554 (#![feature] may not be used on the stable release channel), and without it the use is rejected with E0658 because std::simd is unstable.
std::simd provides portable fixed-width vector types, such as Simd<f32, 8> (aliased as f32x8), with arithmetic operators, lane-wise comparisons that produce masks, and gather and scatter operations, all in safe code that the compiler lowers to whatever the target supports. Because it is unstable, its API can still change, and code using it builds only on nightly. Do not base a design for stable Rust on it.
On stable, third-party crates such as wide provide portable vector types by implementing them with core::arch for each architecture. They are not available in this course, where every solution is a single file using only std, but they are worth evaluating in real projects. When you evaluate one, check whether it chooses instructions at compile time, from the enabled target features, or at run time. A crate that decides at compile time uses only the target's baseline instructions unless you enable more when building.
Check your understanding Given all three tiers, how would you make a numeric kernel fast in a stable-Rust project that you distribute to users? Show answer Hide answer
A reasonable order, stopping as soon as it is fast enough:
1. Measure first. Profile the program and confirm that this kernel is where the time goes. Optimising code that is not the bottleneck adds complexity for no gain.
2. Fix the data layout. Keep the values the kernel reads contiguous in memory, for example one flat Vec<f32> indexed by row * width + col instead of a Vec<Vec<f32>>. Each inner Vec of a Vec<Vec<f32>> is contiguous, so a loop over one row can still vectorise, but every row is a separate allocation, the outer loop follows a pointer per row, and rows can have different lengths. Smaller element types (f32 instead of f64, where the precision is enough) also fit more lanes in each register.
3. Write the loop so LLVM can vectorise it. Several accumulators for float reductions, zip or as_chunks instead of indexing, one assert! up front relating the lengths, and no early exits. Then check the assembly or the loop-vectorize remarks to confirm it worked, and measure again. This step is safe and portable.
4. If it is still the bottleneck, try a stable portable SIMD crate before writing intrinsics. It avoids writing unsafe code and one version per architecture yourself, though you take on a dependency and its choices about which instructions to use.
5. Only then write core::arch intrinsics, for the architectures that matter most, behind runtime feature detection, with the scalar fallback still present and tested against the vector path. Comment each intrinsic call with what it does, because the vendor names are hard to read later.
-C target-cpu=native is not on this list for distributed binaries, for the reason given in tier 2. It is fine for benchmarks and programs that only run on the machine that built them.
Why this is an article and not a problem
core::arch intrinsics compile in a single std-only file, but your submission is compiled and run on your own machine, which might be x86-64 or 64-bit ARM (for example Apple Silicon). An exercise that required x86-64 intrinsics would not compile on ARM, and the reverse. The portable shape, runtime detection with a scalar fallback, runs only one branch on any given machine, and tests that only compare outputs usually cannot tell which branch produced them, so they cannot reliably check that the vector path exists or is correct.
What can be checked anywhere is tier 1: an ordinary loop, written so that LLVM is able to vectorise it, which is also where to start in practice.