Where half precision stops

torch.tensor([65504.], dtype=torch.float16)        # the largest finite value
torch.tensor([65504.], dtype=torch.float16) * 2    # inf
torch.tensor([1e-8], dtype=torch.float16)          # 0.0

float16 has 5 exponent bits, so it covers roughly 6e-8 to 65504. Outside that there is inf at the top and 0 at the bottom, and neither raises.

Why this is a training problem and not a trivia problem

Activations rarely leave the range. Gradients routinely do, at the bottom: a gradient of 1e-8 is entirely ordinary and is exactly zero in float16. Once it is zero it stays zero, and the parameter stops learning.

This is the whole reason loss scaling exists. Multiply the loss by a large constant before backward, so every gradient is scaled up out of the underflow region, then divide the gradients by the same constant before the optimiser step:

(loss * scale).backward()
# then, under no_grad, p.grad /= scale

torch.amp.GradScaler does this and adjusts the scale dynamically, backing off whenever it detects an inf at the top end. Both ends are live.

bfloat16 makes the other trade

float16    5 exponent bits, 10 mantissa    narrow range, more precision
bfloat16   8 exponent bits,  7 mantissa    float32's range, less precision

bfloat16 has the same exponent width as float32, so it does not underflow where float16 does and generally needs no loss scaling. It pays in precision. That is why newer hardware and newer training recipes prefer it.

Your task

def half_info(value: float) -> dict

Return what value becomes in float16:

{"stored": <the float16 value, as a float>, "overflowed": <bool>, "underflowed": <bool>}

Overflow means the value was finite and became infinite. Underflow means it was non-zero and became zero.