Easy End-to-End

Multi-Class Classifier

Evaluate a classifier that assigns each example to one of C classes. The supplied weights and bias produce one score, or logit, per class. Softmax converts each row of scores into probabilities that sum to one. This is a forward pass and loss calculation; do not update the parameters.

Your task

Implement multi_class_classifier(x, W, b, y):

x has shape (N, features), with N >= 1.
W has shape (features, C) and b has shape (C,), with C >= 1.
y has shape (N,) and contains class indices from 0 through C - 1.

Compute z = x @ W + b. Apply softmax across classes within each row, not across examples. Return a dictionary with "probabilities" of shape (N, C) and "loss" as a Python float. Preserve the inputs.

The loss is the mean negative log probability of the correct class. Use each example's own label to select its class, casting labels to integer indices as needed. Do not average across all classes or return argmax labels.

Stable computation

Subtract each row's largest logit before exponentiating. This leaves the probabilities unchanged while avoiding overflow:

shifted = z - row_max(z)
normalizer = row_sum(exp(shifted))
probabilities = exp(shifted) / normalizer
log_probabilities = shifted - log(normalizer)
loss = -mean(log_probabilities[i, y[i]])

Keep the row maximum and normalizer shaped (N, 1) for broadcasting. Compute log probabilities from shifted logits as shown. Taking log of an already rounded probability can give negative infinity when that probability underflows to zero. No epsilon or clipping is needed here.

Example

With x = [[0]], W = [[0, 0]], b = [0, 0] and y = [1], both logits are zero. Return probabilities [[0.5, 0.5]] and loss approximately 0.693147.

For logits [1000, -1000] and target class 1, floating-point probabilities can round to [1, 0], but the loss is still finite, approximately 2000. The stable log-probability calculation preserves that information.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Measure how much probability the model gives the correct class

For a task with mutually exclusive classes, softmax converts the model's scores (logits) into class probabilities. Cross-entropy measures how well those probabilities match a target distribution:

$$L=-\sum_c q_c\log p_c.$$

Here, p contains the predicted probabilities and q contains the target probabilities. For a single correct class y, the target is one at y and zero elsewhere. The loss then becomes -log(p_y).

Suppose the predicted probabilities are [0.8, 0.2]. If the true class is class 0, the loss is about 0.223. If the true class is class 1, the loss is about 1.609. The prediction has not changed; the second loss is larger because the model gives the true class less probability.

Compute log probabilities using log-softmax or log-sum-exp. Taking the logarithm of probabilities after softmax can produce negative infinity if a small probability has already rounded to zero.

Label smoothing gives some probability to other classes

Label smoothing replaces a one-hot target with a less certain target. When the smoothing mass is shared across all C classes, the formula is

$$q=(1-\epsilon)\operatorname{onehot}(y)+\epsilon/C.$$

For two classes and epsilon = 0.1, target [1, 0] becomes [0.95, 0.05]. Some definitions share the smoothing mass only among incorrect classes. Follow the convention in the exercise.

Focal loss gives easy examples less weight

Focal loss multiplies the target's log-loss by (1 - p_t)^gamma, where p_t is the probability assigned to the true class. With positive gamma, an example that already has a high p_t contributes less to training. An optional class weight can further change each example's contribution.

At gamma = 0, the focal factor is one, so the result is cross-entropy with any class weighting still applied. Check the exercise's definition of class weights.

For ordinary mean cross-entropy with one correct class per example, the gradient of the logits is (p - onehot(y)) / N, where N is the batch size. Softmax links all class scores, so all logits can receive a gradient even though the loss is written using only p_y. Average once, and keep the class axis separate from the batch axis.

Detecting runtime…