log1p / expm1 for Small Values

Keep small changes visible

In float32, a tiny change can disappear when it is added to 1. For example, 1 + 1e-8 rounds to 1, so taking its logarithm gives zero. The intended answer is close to 1e-8.

jnp.log1p(x) computes log(1 + x) without first rounding that sum. Similarly, jnp.expm1(x) computes exp(x) - 1 accurately near zero, where subtracting two nearly equal numbers can lose useful digits. Use these functions when your formula contains those expressions.

Worked example

import jax.numpy as jnp

x = jnp.array(1e-8, dtype=jnp.float32)
naive_log = jnp.log(1.0 + x)  # 0.0
stable_log = jnp.log1p(x)    # approximately 1e-8
naive_exp = jnp.exp(x) - 1.0 # 0.0
stable_exp = jnp.expm1(x)    # approximately 1e-8

The size matters: 1 + 1e-7 does not round to 1 in float32. It rounds to the next representable value above 1. That still introduces a noticeable relative error when calculating the small log.

Round trips have limits

In exact real arithmetic, log1p(expm1(x)) equals x. Floating-point arithmetic only approximates that identity. For moderate inputs the result is close to x, but it need not match bit for bit.

For a large positive float32 input such as 100, expm1 overflows to infinity. For a large negative input such as -100, it rounds to exactly -1, and log1p(-1) returns negative infinity. Neither round trip recovers the original finite input.

On its own, real-valued log1p requires an input greater than -1 for a finite result. At -1 it returns negative infinity, and below -1 it returns NaN. These functions improve numerical accuracy; they do not remove domain restrictions, overflow, or rounding.

Floating-point numbers have limited range and precision

A floating-point number stores significant digits and an exponent. The exponent limits its range; the number of significant digits limits its precision. Overflow occurs when a result is too large to represent. Very small values can lose precision or round to zero through underflow. Ordinary rounding can also make nearby values indistinguishable.

Subtracting nearly equal values can leave a result dominated by earlier rounding errors. For tiny x, computing log(1 + x) can fail because 1 + x rounds to 1 before the logarithm is taken. The specialized function log1p(x) avoids that loss. Similarly, expm1(x) computes exp(x) - 1 accurately near zero.

Use formulas with safer intermediate values

A numerically stable formula computes the same mathematical quantity while reducing avoidable rounding error or overflow. For finite, nonempty inputs, log-sum-exp can be computed as

$$\log\sum_i e^{z_i}=m+\log\sum_i e^{z_i-m},\quad m=\max_i z_i.$$

After subtracting m, every exponent is at most zero. This avoids exponentiating large positive numbers. For scores [1000, 1000], the result is 1000 + log(2) without ever computing exp(1000).

Compare with a tolerance appropriate to the task

Floating-point addition is not associative: changing the order of a sum can change its rounded result. Parallel reductions and different matrix-multiplication kernels can therefore produce slightly different answers.

Approximate comparisons usually allow an absolute error near zero and a relative error that scales with the reference value. Choose tolerances suited to the problem's scale and dtype. A loose tolerance should not hide a wrong algorithm.

NaN does not compare equal to itself. Decide how invalid values should be handled rather than assuming an ordinary equality test will catch them. Replacing a NaN after an invalid operation may still leave invalid gradients. When differentiating, avoid the invalid operation itself where possible.