Medium Primitives

Expected Calibration Error

Expected calibration error (ECE) compares a classifier's confidence with how often its predictions are correct. Group predictions with similar confidence, measure the gap in each group, then average those gaps in proportion to the number of examples in each group.

This exercise computes binned predicted-class confidence ECE for a binary classifier. A probability of 0.1 for class 1 means a prediction of class 0 with confidence 0.9, not confidence 0.1.

Inputs and output

Implement expected_calibration_error(probs, labels, num_bins=10).

probs has shape (N,) and contains finite probabilities of class 1 in [0, 1].
labels has shape (N,) and contains true labels 0 or 1, delivered as floats.
num_bins is a positive integer giving the number of equal-width bins.
Return ECE as a Python float in [0, 1]. For N=0, return 0.0 as this exercise's empty-input convention. Leave inputs unchanged.

Compute confidence before binning

For each example:

prediction = 1 if probability > 0.5 else 0
confidence = probability if prediction == 1 else 1 - probability
correct = 1 if prediction == label else 0

A probability of exactly 0.5 predicts class 0 with confidence 0.5. Bin by confidence, not by the original probability of class 1. Confidence therefore lies in [0.5, 1], though the bin grid spans all of [0, 1].

Bin b, numbered from 0, covers [b / num_bins, (b + 1) / num_bins). The final bin includes its right endpoint, 1.0. An exact interior boundary belongs to the bin on its right. For each nonempty bin, compute:

mean_confidence = mean(confidence of examples in the bin)
accuracy = mean(correct for examples in the bin)
contribution = (number in bin / N) * abs(mean_confidence - accuracy)

Sum those contributions. Empty bins contribute zero. Take the absolute gap after averaging within each bin, not separately for each example.

Example

With probs=[0.1, 0.9], labels=[0, 1] and num_bins=2, both predictions are correct and have confidence 0.9. Both enter the upper bin, whose accuracy is 1.0, so ECE is abs(0.9 - 1.0) = 0.1.

With probs=[0, 1] and matching labels [0, 1], both confidence and accuracy are 1, giving ECE 0. A correct class-0 prediction should not be penalized just because its probability of class 1 is low.

Interpreting the result

This is the confidence-versus-accuracy form discussed by Guo et al. (2017), with the explicit left-closed bin convention above. The result depends on the chosen bins and the evaluation sample. A small binned ECE does not prove perfect calibration, and the absolute value does not distinguish overconfidence from underconfidence.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Count the kinds of mistakes the model makes

Accuracy is the fraction of predictions that are correct. A confusion matrix gives more detail by counting each combination of true and predicted class. Check which axis represents the true class before reading the matrix.

For binary classification:

A true positive (TP) is a positive example predicted as positive.
A false positive (FP) is a negative example predicted as positive.
A false negative (FN) is a positive example predicted as negative.

Precision, TP / (TP + FP), asks how many predicted positives were correct. Recall, TP / (TP + FN), asks how many actual positives were found. For TP = 6, FP = 2, and FN = 4, precision is 6/8 = 0.75 and recall is 6/10 = 0.6. F1 combines them using their harmonic mean, 2 * precision * recall / (precision + recall), which is about 0.667 here.

A model can have high accuracy by always predicting the majority class while missing every rare positive. For multiclass F1, macro averaging gives each class equal weight. Micro averaging combines the counts first. Follow the specified policy when a denominator is zero.

AUC measures ranking

ROC-AUC compares positive and negative examples by their scores. It is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with ties usually counted as half a success.

For positive scores [0.9, 0.5] and negative scores [0.4, 0.1], every positive outranks every negative, so AUC is 1. Both classes must be present for this comparison to be meaningful. Handle equal scores explicitly when implementing it.

Calibration measures whether probabilities are trustworthy

Among 100 examples each assigned positive-class probability 0.8, about 80 positives would support calibration for that group. Expected calibration error (ECE) estimates this agreement by grouping predictions into bins and weighting each bin's gap by its number of examples. Bin boundaries and empty-bin handling are part of the definition.

Binary probability calibration compares positive-class probabilities with the frequency of positive labels. Top-label confidence calibration instead compares the probability of the predicted class with how often that prediction is correct. These are different measurements; use the one requested by the exercise.

AUC and calibration answer different questions. A model can rank all positives above all negatives while assigning probabilities that are much too confident.

Detecting runtime…