Easy Primitives

Implement L2 Regularization

L2 regularization adds a penalty for large weights to a training loss. Compute the penalty for the supplied weight vector; do not change the weights or perform an optimizer update.

Your task

Implement l2_regularization(weights, lambda_):

weights is a one-dimensional floating-point tensor.
lambda_ is a finite, nonnegative regularization strength.

Return the scalar tensor:

penalty = (lambda_ / 2) * sum(weights**2)

Sum over every weight. Do not take a square root or divide by the number of weights. Include the factor of one-half: other conventions can absorb it into the strength, but this exercise defines it explicitly.

Keep the result as a zero-dimensional tensor so it can be added to a loss and differentiated. Preserve the input. No data loss is supplied, so return only the regularization penalty, not a combined loss or a gradient vector.

Example

For weights = [1, -2, 3] and lambda_ = 0.1, the squared weights sum to 1 + 4 + 9 = 14. The penalty is 0.1 / 2 * 14 = 0.7. Positive and negative weights contribute equally when their magnitudes are equal. Zero strength or all-zero weights gives zero penalty.

How it affects training

The derivative of this penalty with respect to a weight w is lambda_ * w. Adding it to a data loss therefore adds that term to the weight's gradient. The one-half in the penalty cancels the factor of two from differentiating the square.

This function computes a loss term. An optimizer's weight-decay setting acts during an update and is not an operation to perform here. Return the penalty without modifying the weights.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Predict a numerical value

Regression predicts a number, such as a price or a temperature. A linear model combines the input features using weights and a bias:

$$\hat y=Xw+b.$$

Mean squared error (MSE) averages the squared differences between predictions and targets. For N examples with one target each,

$$L=\frac{1}{N}\sum_i(\hat y_i-y_i)^2.$$

Squaring makes larger errors count more. An error of 4 contributes 16 to the sum, while an error of 2 contributes 4.

The gradient with respect to each prediction is 2 * (prediction - target) / N. For several outputs per example, check whether the loss averages over examples only or over all output elements. The denominator, and therefore the gradient, will differ.

More features can represent curves

Polynomial regression uses features such as [1, x, x²]. The predicted curve can be nonlinear in x while remaining linear in its learned coefficients. Large powers can create very different feature scales, making the fitting problem more sensitive to numerical error and learning-rate choices.

Regularization discourages large weights

Adding the penalty lambda * sum(w²) adds 2 * lambda * w to the weight gradient. Some definitions include a factor of one half, so follow the stated loss rather than assuming a convention.

This penalty is used in ridge regression. It can make fitting more stable when features are strongly correlated, but feature scaling still matters: rescaling a feature changes the coefficient needed to represent the same prediction.

Why squared error favors the mean

For targets [1, 3], predicting 2 for both gives squared errors [1, 1] and MSE 1. Predicting 1 for both gives errors [0, 4] and MSE 2. The mean minimizes squared error for a constant prediction. More generally, when the relevant expectations exist, the best prediction under expected squared loss is the target's conditional mean given the input.

Large outliers can strongly affect this mean and the fitted model. Evaluate on held-out data to check whether a low training loss carries over to new examples.

Detecting runtime…