We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
← Metrics & Evaluation step 1 of 7
Accuracy – Multiclass
Accuracy is the fraction of examples assigned the correct class. When a model returns one score per class, choose the class with the largest score in each row. The index of that score is called the argmax.
For scores [[3, 1, 0], [0, 2, 1], [1, 0, 4]], the predicted classes
are [0, 1, 2]. If the labels are [0, 2, 2], two of three predictions
are correct, so accuracy is 2 / 3, approximately 0.666667.
Your task
Implement accuracy(predictions, labels):
predictionshas shape(N, C), with N > 0 and C > 0. Entries are finite class scores, either logits or probabilities.labelshas shape(N,)and contains whole-number class IDs in[0, C - 1]. The test inputs may store these IDs as floating-point values.- Return a Python float between 0 and 1, not a percentage or a count.
Choose the highest-scoring class independently for each example. If scores tie for the maximum, choose the smallest class index. Compare these IDs with labels, count matches, and divide by N. Softmax is unnecessary: it preserves the ordering of scores within a row in exact arithmetic.
Accuracy counts every example equally and treats all mistakes alike. It can hide poor performance on a rare class: always predicting a class that occurs in 99% of examples gives 99% accuracy while missing every example outside it. Inspect per-class metrics or a confusion matrix when that distinction matters. Class balance alone does not decide which metric best reflects the task.
Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.
Count the kinds of mistakes the model makes
Accuracy is the fraction of predictions that are correct. A confusion matrix gives more detail by counting each combination of true and predicted class. Check which axis represents the true class before reading the matrix.
For binary classification:
- A true positive (
TP) is a positive example predicted as positive. - A false positive (
FP) is a negative example predicted as positive. - A false negative (
FN) is a positive example predicted as negative.
Precision, TP / (TP + FP), asks how many predicted positives were correct.
Recall, TP / (TP + FN), asks how many actual positives were found.
For TP = 6, FP = 2, and FN = 4, precision is 6/8 = 0.75 and recall is
6/10 = 0.6. F1 combines them using their harmonic mean,
2 * precision * recall / (precision + recall), which is about 0.667 here.
A model can have high accuracy by always predicting the majority class while missing every rare positive. For multiclass F1, macro averaging gives each class equal weight. Micro averaging combines the counts first. Follow the specified policy when a denominator is zero.
AUC measures ranking
ROC-AUC compares positive and negative examples by their scores. It is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with ties usually counted as half a success.
For positive scores [0.9, 0.5] and negative scores [0.4, 0.1], every positive
outranks every negative, so AUC is 1. Both classes must be present for this
comparison to be meaningful. Handle equal scores explicitly when implementing it.
Calibration measures whether probabilities are trustworthy
Among 100 examples each assigned positive-class probability 0.8, about 80
positives would support calibration for that group. Expected calibration error
(ECE) estimates this agreement by grouping predictions into bins and weighting
each bin's gap by its number of examples. Bin boundaries and empty-bin handling
are part of the definition.
Binary probability calibration compares positive-class probabilities with the frequency of positive labels. Top-label confidence calibration instead compares the probability of the predicted class with how often that prediction is correct. These are different measurements; use the one requested by the exercise.
AUC and calibration answer different questions. A model can rank all positives above all negatives while assigning probabilities that are much too confident.
Your code, saved here
Sign in to access your accepted solutions and saved draft history. You can keep reading the lesson without an account.
Sign in to your accountStuck?
JAX reference solution
Sign in to track your attempts toward unlocking the reference solution.