Easy Research

Label Smoothing

Implement label_smoothing: convert class IDs into probability distributions by mixing each one-hot target with the uniform distribution over all classes. This gives some target probability to other classes instead of assigning the entire mass to one class. The function returns targets, not predictions or loss.

Contract

labels: shape (B,), integer-valued class IDs delivered as a floating-point tensor. Each ID is in [0, K).
n_classes: a positive integer K.
epsilon: a finite smoothing coefficient in [0, 1].

Return shape (B, K), preserving the floating-point dtype and device of labels. Leave labels unchanged. B may be zero; then return an empty (0, K) tensor.

For each row, put epsilon/K in every column, then add 1-epsilon at its true class ID. Equivalently:

true-class probability = 1 - epsilon + epsilon/K
other-class probability = epsilon/K

The uniform part includes the true class. Do not distribute epsilon only among the K-1 incorrect classes. Every row sums to one, apart from floating-point rounding. Repeated class IDs produce repeated rows.

For labels=[1,0], K=3 and epsilon=0.3, return:

[[0.1, 0.8, 0.1],
 [0.8, 0.1, 0.1]]

With epsilon=0, the result is one-hot. With epsilon=1, each row is uniform. With K=1, every nonempty row is [1] for any allowed epsilon. Build the target tensor directly; do not call a loss function.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Measure how much probability the model gives the correct class

For a task with mutually exclusive classes, softmax converts the model's scores (logits) into class probabilities. Cross-entropy measures how well those probabilities match a target distribution:

$$L=-\sum_c q_c\log p_c.$$

Here, p contains the predicted probabilities and q contains the target probabilities. For a single correct class y, the target is one at y and zero elsewhere. The loss then becomes -log(p_y).

Suppose the predicted probabilities are [0.8, 0.2]. If the true class is class 0, the loss is about 0.223. If the true class is class 1, the loss is about 1.609. The prediction has not changed; the second loss is larger because the model gives the true class less probability.

Compute log probabilities using log-softmax or log-sum-exp. Taking the logarithm of probabilities after softmax can produce negative infinity if a small probability has already rounded to zero.

Label smoothing gives some probability to other classes

Label smoothing replaces a one-hot target with a less certain target. When the smoothing mass is shared across all C classes, the formula is

$$q=(1-\epsilon)\operatorname{onehot}(y)+\epsilon/C.$$

For two classes and epsilon = 0.1, target [1, 0] becomes [0.95, 0.05]. Some definitions share the smoothing mass only among incorrect classes. Follow the convention in the exercise.

Focal loss gives easy examples less weight

Focal loss multiplies the target's log-loss by (1 - p_t)^gamma, where p_t is the probability assigned to the true class. With positive gamma, an example that already has a high p_t contributes less to training. An optional class weight can further change each example's contribution.

At gamma = 0, the focal factor is one, so the result is cross-entropy with any class weighting still applied. Check the exercise's definition of class weights.

For ordinary mean cross-entropy with one correct class per example, the gradient of the logits is (p - onehot(y)) / N, where N is the batch size. Softmax links all class scores, so all logits can receive a gradient even though the loss is written using only p_y. Average once, and keep the class axis separate from the batch axis.

Detecting runtime…

Label Smoothing

Contract

Measure how much probability the model gives the correct class

Label smoothing gives some probability to other classes

Focal loss gives easy examples less weight

Your code, saved here

Stuck?

PyTorch reference solution