Medium Primitives

Precision, Recall & F1

Precision and recall answer different questions for each class:

Precision: of the examples predicted as this class, how many were correct?
Recall: of the examples truly in this class, how many were found?

For a chosen class, a true positive (TP) is a correct prediction of that class. A false positive (FP) is a prediction of that class with another true label. A false negative (FN) is a true example of that class predicted as something else.

precision = TP / (TP + FP)

recall = TP / (TP + FN)

F1 = 2 * precision * recall / (precision + recall)

F1 is the harmonic mean of precision and recall. It is zero if either is zero. It summarizes the two rates, but does not replace choosing how much false positives and false negatives matter for a particular task.

For labels [0, 0, 1, 1] and predictions [0, 1, 1, 1], class 0 has precision 1, recall 0.5, and F1 about 0.666667. Class 1 has precision about 0.666667, recall 1, and F1 0.8. Their macro-F1 is the mean of those two F1 values, approximately 0.733333.

Your task

Implement precision_recall_f1(predictions, labels, num_classes):

predictions and labels are class IDs with matching shape (N,), not score vectors. Whole-number IDs lie in [0, C - 1] and may be stored as floating-point values by the test input.
num_classes is a positive integer C. Compute results for all C classes.
Whenever a metric's denominator is zero, return 0 for that metric. Thus a class absent from both predictions and labels has P=R=F1=0.

Compute each metric separately per class, then take its unweighted mean over all C classes, including absent ones. These are macro-averages. Macro-F1 is the mean of the per-class F1 values, not the harmonic mean of macro-precision and macro-recall. Do not weight by class frequency. For empty inputs, every entry of the result is zero.

Return a floating-point tensor of shape (3, C + 1):

Row	First C columns, in class-ID order	Last column
0	Per-class precision	Macro-precision
1	Per-class recall	Macro-recall
2	Per-class F1	Macro-F1

A confusion matrix can help: with true labels in rows and predictions in columns, TP is the diagonal, TP+FP is a column sum, and TP+FN is a row sum. You may also count those quantities directly.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Count the kinds of mistakes the model makes

Accuracy is the fraction of predictions that are correct. A confusion matrix gives more detail by counting each combination of true and predicted class. Check which axis represents the true class before reading the matrix.

For binary classification:

A true positive (TP) is a positive example predicted as positive.
A false positive (FP) is a negative example predicted as positive.
A false negative (FN) is a positive example predicted as negative.

Precision, TP / (TP + FP), asks how many predicted positives were correct. Recall, TP / (TP + FN), asks how many actual positives were found. For TP = 6, FP = 2, and FN = 4, precision is 6/8 = 0.75 and recall is 6/10 = 0.6. F1 combines them using their harmonic mean, 2 * precision * recall / (precision + recall), which is about 0.667 here.

A model can have high accuracy by always predicting the majority class while missing every rare positive. For multiclass F1, macro averaging gives each class equal weight. Micro averaging combines the counts first. Follow the specified policy when a denominator is zero.

AUC measures ranking

ROC-AUC compares positive and negative examples by their scores. It is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with ties usually counted as half a success.

For positive scores [0.9, 0.5] and negative scores [0.4, 0.1], every positive outranks every negative, so AUC is 1. Both classes must be present for this comparison to be meaningful. Handle equal scores explicitly when implementing it.

Calibration measures whether probabilities are trustworthy

Among 100 examples each assigned positive-class probability 0.8, about 80 positives would support calibration for that group. Expected calibration error (ECE) estimates this agreement by grouping predictions into bins and weighting each bin's gap by its number of examples. Bin boundaries and empty-bin handling are part of the definition.

Binary probability calibration compares positive-class probabilities with the frequency of positive labels. Top-label confidence calibration instead compares the probability of the predicted class with how often that prediction is correct. These are different measurements; use the one requested by the exercise.

AUC and calibration answer different questions. A model can rank all positives above all negatives while assigning probabilities that are much too confident.

Detecting runtime…

Precision, Recall & F1

Your task

Count the kinds of mistakes the model makes

AUC measures ranking

Calibration measures whether probabilities are trustworthy

Your code, saved here

Stuck?

PyTorch reference solution