Skip to content

← Reductions and dim step 6 of 7

Medium Primitives

Sum float32 values using float64 accumulation

Jump to the task

A sum can contain two kinds of rounding error: the error already present in the stored inputs, and additional error introduced during reduction. Using a wider reduction dtype can reduce the second, but cannot undo the first.

import torch

x = torch.full((1_000_000,), 0.1, dtype=torch.float32)
print(x[0].item())                    # 0.10000000149011612
print(x.sum(dtype=torch.float64).item()) # 100000.00149011612

The mathematical sum of one million exact decimal values of 0.1 is 100000.0. These inputs are float32 approximations of 0.1, so even their float64 sum differs from that decimal result.

Rounding during the sum

Float32 has 24 bits of significand precision, including the implicit leading bit for normal values. It cannot preserve every small change to a large partial sum. A reduction can lose information as it combines values:

values = torch.tensor([16777216.0, 1.0, -16777216.0], dtype=torch.float32)
print(values.sum().item())                    # 0.0 on CPU
print(values.sum(dtype=torch.float64).item())  # 1.0

The inputs in this second example are exactly representable. The difference comes from the reduction. Addition order matters: a parallel reduction need not round like a Python loop adding one element at a time. Results and error can vary with input size, order and backend; error does not necessarily grow monotonically with the number of terms.

Request a wider reduction

x.sum(dtype=torch.float64) requests a float64 reduction and result without modifying x. This is clearer than explicitly writing x.double().sum() when the wider tensor is needed only for the sum.

The original input stays float32, but this is not a guarantee of zero extra memory or negligible runtime cost. Internal conversion, temporary storage and float64 throughput depend on the implementation and device. Wider precision also does not make every sum exact.

For metrics, losses and other reductions, choose precision based on the required accuracy and measured cost. Do not assume that every low-precision operation uses the same internal accumulation dtype.

Your task

def accurate_sum(x: torch.Tensor) -> torch.Tensor

Given a CPU float32 tensor, return its sum as a zero-dimensional float64 tensor. Request float64 in the reduction rather than explicitly converting the whole tensor first. Sum the values actually stored in x; do not try to reconstruct the decimal values used to create it.