Medium Primitives

Implement Cross-Entropy Loss

For classification with several classes, cross-entropy rewards assigning high probability to the correct class. For each example, select the probability of its target class and take its negative natural logarithm. Then average over the examples.

Suppose the probability rows are [[0.7, 0.2, 0.1], [0.1, 0.8, 0.1]] and the target class indices are [0, 1]. Select 0.7 from the first row and 0.8 from the second. The loss is (-log(0.7) - log(0.8)) / 2, approximately 0.289909.

Each example contributes one selected probability. Do not average over all classes or choose the most likely class instead of the target class.

Your task

Implement cross_entropy_loss(probs, targets):

probs is a floating-point tensor of shape (N, C), with N > 0 and C > 0. Each row contains probabilities in [0, 1] summing to 1.
targets is an integer tensor of shape (N,). Each value is a zero-based class index from 0 through C - 1.
Return a scalar tensor containing the mean negative log probability of the target classes.

Before taking logarithms, clamp probabilities to [1e-7, 1.0]. Do not add epsilon to all values or renormalize after clamping. A target-class probability of zero therefore produces a finite loss of -log(1e-7).

Inputs are already probabilities. In contrast, PyTorch's built-in cross_entropy expects logits, so passing these inputs to that function would compute a different objective. Use basic tensor operations here.

With its default target of 1, the plot shows -log(p) for the probability assigned to a correct class, away from the clamped endpoint. The exercise averages this quantity across rows.

Loading visualization…

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Measure how much probability the model gives the correct class

For a task with mutually exclusive classes, softmax converts the model's scores (logits) into class probabilities. Cross-entropy measures how well those probabilities match a target distribution:

$$L=-\sum_c q_c\log p_c.$$

Here, p contains the predicted probabilities and q contains the target probabilities. For a single correct class y, the target is one at y and zero elsewhere. The loss then becomes -log(p_y).

Suppose the predicted probabilities are [0.8, 0.2]. If the true class is class 0, the loss is about 0.223. If the true class is class 1, the loss is about 1.609. The prediction has not changed; the second loss is larger because the model gives the true class less probability.

Compute log probabilities using log-softmax or log-sum-exp. Taking the logarithm of probabilities after softmax can produce negative infinity if a small probability has already rounded to zero.

Label smoothing gives some probability to other classes

Label smoothing replaces a one-hot target with a less certain target. When the smoothing mass is shared across all C classes, the formula is

$$q=(1-\epsilon)\operatorname{onehot}(y)+\epsilon/C.$$

For two classes and epsilon = 0.1, target [1, 0] becomes [0.95, 0.05]. Some definitions share the smoothing mass only among incorrect classes. Follow the convention in the exercise.

Focal loss gives easy examples less weight

Focal loss multiplies the target's log-loss by (1 - p_t)^gamma, where p_t is the probability assigned to the true class. With positive gamma, an example that already has a high p_t contributes less to training. An optional class weight can further change each example's contribution.

At gamma = 0, the focal factor is one, so the result is cross-entropy with any class weighting still applied. Check the exercise's definition of class weights.

For ordinary mean cross-entropy with one correct class per example, the gradient of the logits is (p - onehot(y)) / N, where N is the batch size. Softmax links all class scores, so all logits can receive a gradient even though the loss is written using only p_y. Average once, and keep the class axis separate from the batch axis.

Detecting runtime…

Implement Cross-Entropy Loss

Your task

Measure how much probability the model gives the correct class

Label smoothing gives some probability to other classes

Focal loss gives easy examples less weight

Your code, saved here

Stuck?

PyTorch reference solution