Medium Primitives

AUC-ROC

ROC AUC measures how well scores rank positive examples above negative examples, without choosing a classification threshold. Each positive-negative pair contributes 1 if the positive score is higher, 0.5 if the scores tie, and 0 if the positive score is lower.

Your task

Implement auc_roc(scores, labels).

scores is a finite floating-point vector of shape (N,). Higher scores indicate stronger evidence for class 1. Scores need not be probabilities: negative values and values above 1 are valid.
labels has shape (N,), with true labels 0 or 1, delivered as floats.
Return a Python float in [0, 1], leaving both inputs unchanged.

Let n_pos and n_neg be the numbers of positive and negative examples. Compare every positive score with every negative score, then compute:

AUC = (number of wins + 0.5 * number of ties) / (n_pos * n_neg)

Use exact equality for ties; do not round scores before comparing them. The denominator counts pairs, not examples. Do not threshold the scores at 0.5 or compute classification accuracy.

If either class is absent, including an empty input, return 0.0 by this exercise's convention. ROC AUC is undefined in that situation, so this return value should not be interpreted as evidence of a completely reversed ranking.

Example

For positive scores [0.9, 0.5] and negative scores [0.5, 0.1], the four pairs contribute [1, 1, 0.5, 1]. AUC is 3.5 / 4 = 0.875.

If both classes are present and every score ties, AUC is 0.5. If every positive scores above every negative, AUC is 1. If every positive scores below every negative, AUC is 0.

Connection to the ROC curve

The ROC curve plots the fraction of negatives predicted positive (false positive rate) on the horizontal axis and the fraction of positives predicted positive (true positive rate) on the vertical axis, as the score threshold moves from above the largest score to below the smallest.

The pairwise formula gives the trapezoidal area under that curve when equal scores cross the threshold together. You may use pair counting, average ranks for tied scores, or grouped-threshold integration. Processing tied examples one at a time in an arbitrary order can give the wrong area.

AUC measures ranking, not probability calibration or performance at a specific threshold. Class imbalance does not change the pair-weighting rule, but a high AUC alone does not tell you the precision or number of false positives at an operating threshold.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Count the kinds of mistakes the model makes

Accuracy is the fraction of predictions that are correct. A confusion matrix gives more detail by counting each combination of true and predicted class. Check which axis represents the true class before reading the matrix.

For binary classification:

A true positive (TP) is a positive example predicted as positive.
A false positive (FP) is a negative example predicted as positive.
A false negative (FN) is a positive example predicted as negative.

Precision, TP / (TP + FP), asks how many predicted positives were correct. Recall, TP / (TP + FN), asks how many actual positives were found. For TP = 6, FP = 2, and FN = 4, precision is 6/8 = 0.75 and recall is 6/10 = 0.6. F1 combines them using their harmonic mean, 2 * precision * recall / (precision + recall), which is about 0.667 here.

A model can have high accuracy by always predicting the majority class while missing every rare positive. For multiclass F1, macro averaging gives each class equal weight. Micro averaging combines the counts first. Follow the specified policy when a denominator is zero.

AUC measures ranking

ROC-AUC compares positive and negative examples by their scores. It is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with ties usually counted as half a success.

For positive scores [0.9, 0.5] and negative scores [0.4, 0.1], every positive outranks every negative, so AUC is 1. Both classes must be present for this comparison to be meaningful. Handle equal scores explicitly when implementing it.

Calibration measures whether probabilities are trustworthy

Among 100 examples each assigned positive-class probability 0.8, about 80 positives would support calibration for that group. Expected calibration error (ECE) estimates this agreement by grouping predictions into bins and weighting each bin's gap by its number of examples. Bin boundaries and empty-bin handling are part of the definition.

Binary probability calibration compares positive-class probabilities with the frequency of positive labels. Top-label confidence calibration instead compares the probability of the predicted class with how often that prediction is correct. These are different measurements; use the one requested by the exercise.

AUC and calibration answer different questions. A model can rank all positives above all negatives while assigning probabilities that are much too confident.

Detecting runtime…