Skip to content

← Dtypes and Numerics step 4 of 6

Medium Primitives

Where half precision stops

Jump to the task

Converting a tensor to float16 can change both its precision and the range of values it can store. Large magnitudes can become infinite, while small nonzero values can round to zero.

import torch

x = torch.tensor([65504.0, 131008.0, 1e-8], dtype=torch.float32)
print(x.to(torch.float16).tolist())  # [65504.0, inf, 0.0]

Range is not a pair of exact cutoffs

The largest finite float16 value is 65504. The smallest positive normal value is about 6.10e-5; smaller subnormal values extend down to about 5.96e-8. Float16 also represents zero and negative values.

Conversion rounds to representable values. A number just above 65504 can round back to it, and a positive number below the smallest subnormal can round up to that subnormal rather than to zero:

near_edges = torch.tensor([65505.0, 4e-8], dtype=torch.float32)
print(near_edges.to(torch.float16).tolist())
# [65504.0, 5.960464477539063e-08]

Why this matters during training

If a gradient contribution rounds to zero, that contribution is lost. This does not mean the parameter can never learn again: later gradients can differ, and optimizer state may also affect updates.

Loss scaling can reduce gradient underflow by scaling the loss before backward and unscaling gradients before the optimizer update. It cannot recover information already lost earlier in the computation, and a scale that is too large can cause overflow. torch.amp.GradScaler checks for nonfinite gradients, can skip affected updates and adjusts the scale.

Bfloat16 uses eight exponent bits instead of float16's five, giving it a much wider range, comparable to float32. It has fewer fraction bits, so nearby values can round together more readily. It often avoids the need for loss scaling, but is not immune to underflow, overflow or rounding error.

Your task

def half_info(value: float) -> dict

Create a float32 tensor from the finite input value, then convert it to float16. Return a dictionary with:

  • "stored": the converted value as a Python float, including signed infinity.
  • "overflowed": whether a finite input became infinite.
  • "underflowed": whether a nonzero input became zero.

Here underflowed specifically means rounding to zero; it does not flag nonzero subnormals. The test harness represents either infinity as the text "inf" for serialization, but your function should return the float itself.