Eval Loop with Metrics

Evaluate a fixed binary classifier over several batches. Report its mean binary cross-entropy loss and accuracy across all examples. This function only measures performance; it does not train or update the weights.

Your task

Implement eval_loop(weights, xs, ys) for floating-point inputs:

weights has shape (d,).
xs has shape (K, B, d): K batches of B examples each.
ys has shape (K, B) and contains binary labels 0 or 1.

K, B and d are positive. These are stacked arrays with equal batch sizes, not a list of variable-sized batches. For each batch, compute logits z = xs[k] @ weights, with no bias or activation layer.

Loss and predictions

Compute binary cross-entropy directly from logits using natural logs:

loss_per_example = max(z, 0) - z * y + log1p(exp(-abs(z)))

The exponential receives a nonpositive argument, avoiding overflow. This also avoids taking the log of a sigmoid probability that has rounded to zero or one. Do not add epsilon or clamp the loss.

Predict class 1 when z > 0, otherwise class 0. In particular, a zero logit predicts class 0. This is the decision boundary for sigmoid probability strictly above 0.5; comparing logits directly avoids rounding very small positive scores to an apparent probability tie.

Aggregate across examples

Accumulate the sum of individual losses, the number of correct predictions and the number of examples. Divide the first two totals by K * B once at the end. Return a tensor of shape (2,) in this order:

[mean_bce_loss, accuracy]

Accuracy is a fraction from 0 to 1, not a percentage. Preserve all inputs. In PyTorch, use torch.no_grad() to avoid recording an evaluation graph; simply choosing not to update weights does not disable gradient recording.

For these equal-sized batches, averaging batch means would give the same answer. Summing individual contributions is also the pattern to use when extending an evaluation loop to unequal batches: each example must have equal weight, regardless of its batch.

Example

For weights = [0], xs = [[[1], [2]]] and ys = [[0, 1]], both logits are zero. Each loss is log(2), and only the first prediction is correct. Return approximately [0.693147, 0.5].

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Set the learning rate and control gradient size

A learning-rate schedule changes the step size during training. Check its exact indexing: a warmup may start at zero or at a first positive fraction. Test the initial value, the warmup boundary, the decay endpoint, and any later steps. The schedule's name alone does not specify those choices.

Global norm clipping scales all gradients by the same factor:

$$\min\left(1,\frac{c}{\|g\|}\right),$$

where g represents all parameter gradients together and c is the norm limit. For scalar gradients 3 and 4, the global norm is 5. Clipping to 2.5 multiplies both by 0.5, giving [1.5, 2]. Clipping each tensor separately can change their relative direction. Handle zero norm without dividing by zero.

Keep a smoothed copy of the parameters

An exponential moving average (EMA) updates a separate parameter copy:

$$\bar\theta_t=\beta\bar\theta_{t-1}+(1-\beta)\theta_t.$$

For old average 2, new parameter 4, and beta = 0.9, the new average is 0.9*2 + 0.1*4 = 2.2. A larger beta retains more of the past. Initialization and whether EMA runs before or after the optimizer step affect the result.

Use validation to decide when to stop

Evaluate using the intended inference behavior without changing model parameters. For mean per-example loss, weight each batch's contribution by its example count. Giving unequal batches equal weight changes the overall mean.

Early stopping monitors a validation metric and stops after a specified number of observations without improvement. Define ties, the minimum improvement, and patience precisely. Save an independent copy of the best parameters: the final step may not be the best one. Keep the test set out of the stopping decision so it remains an independent evaluation.

Detecting runtime…