Easy Primitives

Implement Tanh

Hyperbolic tangent, or tanh, maps negative inputs to negative outputs and positive inputs to positive outputs. It is zero at zero and approaches -1 or 1 as the input magnitude grows:

tanh(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))

For [-1.0, 0.0, 1.0], the result is approximately [-0.761594, 0.0, 0.761594]. Unlike sigmoid, the output is centered around zero.

Compute the same function without overflow

The direct formula can produce infinity divided by infinity for large finite inputs. Divide its numerator and denominator by the larger exponential instead. If z = exp(-2 * abs(x)), the output's magnitude is (1 - z) / (1 + z). Give that magnitude the sign of x.

Near zero, subtracting z from one can lose precision. The function expm1(t) computes exp(t) - 1 accurately for small t. With u = expm1(-2 * abs(x)), the magnitude becomes -u / (2 + u).

You can construct the nonpositive exponent with where: choose -2x for nonnegative x and 2x otherwise. Compute expm1 once, then choose the output sign. This avoids large positive exponential arguments while retaining small outputs near zero.

Your task

Implement tanh(x) elementwise for a floating-point tensor of any shape. Return the same shape. Do not use torch.tanh, jnp.tanh or jax.nn.tanh; use basic arithmetic, expm1 and tensor selection.

The tests include large finite inputs and small inputs near zero. Avoid NaN from overflowing exponentials. In exact arithmetic the result lies strictly between -1 and 1 for finite x; floating-point results can round to either endpoint. This exercise checks forward values.

Loading visualization…

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Nonlinear activations let a network learn more than a linear map

An affine layer applies weights and adds a bias. Two such layers without a nonlinear activation between them can be combined into one:

$$(xW_1+b_1)W_2+b_2=x(W_1W_2)+(b_1W_2+b_2).$$

An activation adds an input-dependent response that cannot generally be absorbed into a single set of weights and biases.

Common elementwise activations

ReLU, max(0, x), keeps positive values and sets negative values to zero. For example, [-2, 0, 3] becomes [0, 0, 3]. Its derivative is zero for negative inputs and one for positive inputs. At zero, the ordinary derivative does not exist, so an implementation needs a convention.

Leaky ReLU uses a small slope on the negative side instead of setting it to zero. Sigmoid maps finite real inputs to probabilities between zero and one in exact arithmetic. Tanh maps them between negative one and one. For both sigmoid and tanh, large input magnitudes lead to small derivatives, which can slow learning.

SiLU and GELU use smooth gates

SiLU is x * sigmoid(x). Its derivative includes both the direct change in x and the change in the sigmoid factor.

GELU is x * Phi(x), where Phi(x) is the probability that a standard normal random variable is at most x. A common implementation approximates this using tanh. The approximation is close but not identical, so use the version specified in the exercise.

These activations act independently on each element and preserve shape. Softmax instead combines scores along an axis so the resulting probabilities sum to one. Applying sigmoid to every class score is not the same operation.

Use stable formulas where needed. For example, computing sigmoid directly as 1 / (1 + exp(-x)) can overflow in the exponential for a large negative x, even though the desired output is close to zero.

Detecting runtime…

Implement Tanh

Compute the same function without overflow

Your task

Nonlinear activations let a network learn more than a linear map

Common elementwise activations

SiLU and GELU use smooth gates

Your code, saved here

Stuck?

PyTorch reference solution