Compare floating-point tensors with tolerances

Floating-point calculations can introduce rounding differences even when the mathematical results are equal. For example, on CPU with float32:

import torch

x = torch.tensor(2.0, dtype=torch.float32).sqrt() ** 2
print(x.item())                              # 1.9999998807907104
print(bool(x == 2.0))                        # False
print(torch.allclose(x, torch.tensor(2.0)))  # True

Float32 cannot represent the square root of 2 exactly. Squaring its stored approximation can therefore give a nearby value rather than exactly 2. The size of rounding differences depends on the calculation, dtype and backend.

Absolute and relative tolerance

For finite values, torch.allclose(a, b) checks every element against:

abs(a - b) <= atol + rtol * abs(b)

atol is an absolute allowance in the same units as the values. rtol scales the allowance with the magnitude of the reference b. Since the formula uses b, swapping the arguments can change the answer.

The defaults are atol=1e-8 and rtol=1e-5. These are comparison defaults, not universal error bounds for float32. Choose tolerances for the accuracy your application needs and the errors you expect.

a = torch.tensor([1e-9])
b = torch.tensor([0.0])
print(torch.allclose(a, b, atol=1e-8, rtol=0.0))  # True
print(torch.allclose(a, b, atol=0.0, rtol=1e-5))  # False

Here the reference is zero, so the relative term contributes nothing. Setting either tolerance to zero can be intentional: absolute-only and relative-only comparisons are both useful when they match the requirement.

Choose the comparison you need

torch.allclose returns one Python boolean. torch.isclose returns a boolean tensor identifying which elements are close. NaNs do not compare close by default; matching infinities are handled separately from the finite formula above.

Exact equality is appropriate when exact agreement is the requirement, including comparisons of indices, labels or deliberately preserved values. Some floating-point arithmetic is exact too. Do not add a tolerance merely because an input has a floating dtype, or assume all computed floats differ.

For test assertions, torch.testing.assert_close provides useful diagnostics and dtype-dependent defaults. Review those tolerances for your calculation rather than treating them as a guarantee of acceptable accuracy.

Floating-point numbers have limited range and precision

A floating-point number stores significant digits and an exponent. The exponent limits its range; the number of significant digits limits its precision. Overflow occurs when a result is too large to represent. Very small values can lose precision or round to zero through underflow. Ordinary rounding can also make nearby values indistinguishable.

Subtracting nearly equal values can leave a result dominated by earlier rounding errors. For tiny x, computing log(1 + x) can fail because 1 + x rounds to 1 before the logarithm is taken. The specialized function log1p(x) avoids that loss. Similarly, expm1(x) computes exp(x) - 1 accurately near zero.

Use formulas with safer intermediate values

A numerically stable formula computes the same mathematical quantity while reducing avoidable rounding error or overflow. For finite, nonempty inputs, log-sum-exp can be computed as

$$\log\sum_i e^{z_i}=m+\log\sum_i e^{z_i-m},\quad m=\max_i z_i.$$

After subtracting m, every exponent is at most zero. This avoids exponentiating large positive numbers. For scores [1000, 1000], the result is 1000 + log(2) without ever computing exp(1000).

Compare with a tolerance appropriate to the task

Floating-point addition is not associative: changing the order of a sum can change its rounded result. Parallel reductions and different matrix-multiplication kernels can therefore produce slightly different answers.

Approximate comparisons usually allow an absolute error near zero and a relative error that scales with the reference value. Choose tolerances suited to the problem's scale and dtype. A loose tolerance should not hide a wrong algorithm.

NaN does not compare equal to itself. Decide how invalid values should be handled rather than assuming an ordinary equality test will catch them. Replacing a NaN after an invalid operation may still leave invalid gradients. When differentiating, avoid the invalid operation itself where possible.