Medium Research

Focal Loss

Focal loss reduces the contribution of easy classification examples. It starts with cross-entropy and multiplies it by a factor that is small when the model already assigns high probability to the correct label.

For a binary target, let p be the predicted probability of label 1. The probability of the correct label, pt, is p for target 1 and 1 - p for target 0. The class weight, at, is alpha for target 1 and 1 - alpha for target 0.

The focal term is (1 - pt) ** gamma. With gamma = 2, a correct-label probability of 0.9 gives a multiplier of 0.01, while 0.5 gives 0.25. Thus easy examples contribute less, relative to hard examples. alpha separately controls the relative weighting of the two labels.

Your task

Implement focal_loss(probs, targets, alpha, gamma) for nonempty 1D tensors of matching shape. probs contains probabilities in [0, 1]; targets contains binary labels 0 or 1. alpha is a float in [0, 1], and gamma is a nonnegative float.

Compute the mean of

-at * (1 - pt) ** gamma * log(pt + 1e-8)

using natural logarithms. The added 1e-8 is this exercise's explicit finite-log convention. Add it only inside the logarithm; do not change the focal multiplier or substitute probability clamping.

For probs = [0.9], targets = [1], alpha = 1, and gamma = 2, the result is approximately 0.0010536. When gamma = 0, the focal multiplier is 1, leaving class-weighted cross-entropy with the stated epsilon. This is not unweighted BCE: for example, alpha = 1 ignores all target-0 examples.

Return a Python float. Rounding to seven decimal places is optional. For training, keep the result as a tensor instead so gradients remain available. The original focal-loss paper explains its use for dense object detection.

For comparison, the plot below shows ordinary binary cross-entropy. It does not include focal loss's class weight or focusing factor, and uses the chart's own probability-clamping rule.

Loading visualization…

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Measure how much probability the model gives the correct class

For a task with mutually exclusive classes, softmax converts the model's scores (logits) into class probabilities. Cross-entropy measures how well those probabilities match a target distribution:

$$L=-\sum_c q_c\log p_c.$$

Here, p contains the predicted probabilities and q contains the target probabilities. For a single correct class y, the target is one at y and zero elsewhere. The loss then becomes -log(p_y).

Suppose the predicted probabilities are [0.8, 0.2]. If the true class is class 0, the loss is about 0.223. If the true class is class 1, the loss is about 1.609. The prediction has not changed; the second loss is larger because the model gives the true class less probability.

Compute log probabilities using log-softmax or log-sum-exp. Taking the logarithm of probabilities after softmax can produce negative infinity if a small probability has already rounded to zero.

Label smoothing gives some probability to other classes

Label smoothing replaces a one-hot target with a less certain target. When the smoothing mass is shared across all C classes, the formula is

$$q=(1-\epsilon)\operatorname{onehot}(y)+\epsilon/C.$$

For two classes and epsilon = 0.1, target [1, 0] becomes [0.95, 0.05]. Some definitions share the smoothing mass only among incorrect classes. Follow the convention in the exercise.

Focal loss gives easy examples less weight

Focal loss multiplies the target's log-loss by (1 - p_t)^gamma, where p_t is the probability assigned to the true class. With positive gamma, an example that already has a high p_t contributes less to training. An optional class weight can further change each example's contribution.

At gamma = 0, the focal factor is one, so the result is cross-entropy with any class weighting still applied. Check the exercise's definition of class weights.

For ordinary mean cross-entropy with one correct class per example, the gradient of the logits is (p - onehot(y)) / N, where N is the batch size. Softmax links all class scores, so all logits can receive a gradient even though the loss is written using only p_y. Average once, and keep the class axis separate from the batch axis.

Detecting runtime…

Focal Loss

Your task

Measure how much probability the model gives the correct class

Label smoothing gives some probability to other classes

Focal loss gives easy examples less weight

Your code, saved here

Stuck?

PyTorch reference solution