The sum that drifts

Add 0.1 to itself a million times in float32:

float32 sum   100000.015625
float64 sum   100000.00149...
exact         100000.0

The error is not in the last decimal place. It is 0.0156, and it grows with the number of terms.

Where it comes from

float32 has 24 bits of mantissa, so around 7 significant decimal digits. Once the running total reaches 100,000, the smallest change it can represent is about 0.0078. Adding 0.1 to it is fine, but the rounding at each step is a fraction of that, and a million roundings in the same direction do not cancel.

This is why the accumulator’s dtype matters more than the data’s. The values being summed are perfectly representable; the total is where precision is lost.

The fix

Accumulate wider than you store:

x.sum(dtype=torch.float64)

The tensor stays float32; only the accumulator is float64. That costs nothing in memory and almost nothing in time, and it is what dtype= on a reduction is for.

Where this actually matters

• Loss and metric accumulation over an epoch. Thousands of batches summed in float32 is exactly this situation.
• Mean and variance over large tensors, which is why normalisation layers accumulate in float32 even when the activations are float16.
• Anything in float16 or bfloat16, where the mantissa is 11 or 8 bits and the same effect arrives after hundreds of terms rather than millions.

PyTorch already does the right thing inside many of its own reductions. It cannot do it for the loop you wrote.

Your task

def accurate_sum(x: torch.Tensor) -> torch.Tensor

x is a float32 tensor. Return its sum accumulated in float64, as a float64 scalar tensor. Do not convert the whole tensor first: the point is the accumulator, not the storage.