We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
← Normalization step 6 of 6
Feature Normalization Pipeline
Feature normalization puts columns with different units or scales on more comparable numerical scales. This exercise offers two transformations, with separate statistics for every column of the input matrix.
Z-score standardization subtracts the column mean and divides by its
population standard deviation plus epsilon. For a column [1, 3, 5],
the mean is 3 and the standard deviation is sqrt(8 / 3). The result is
approximately [-1.224745, 0, 1.224745].
Min-max scaling subtracts the column minimum and divides by its range
plus epsilon. For [0, 5, 10], the result is approximately [0, 0.5, 1].
Epsilon slightly reduces the maximum from 1 in exact arithmetic, though
floating-point rounding can hide that difference.
Your task
Implement normalize_features(x, method):
xis a floating-point matrix of shape(N, D), with positive dimensions. Rows are examples and columns are features.methodis exactly"zscore"or"minmax".- Return a floating-point tensor of the same shape. Compute statistics from the supplied matrix; no stored statistics or fitting step is required.
Use a fixed eps = 1e-7 and reduce over axis 0:
- For
"zscore", return(x - mean) / (std + eps), where std is the square root of the mean squared deviation. Divide variance by N, not N-1. Add epsilon to the standard deviation, outside its square root. - For
"minmax", return(x - minimum) / (maximum - minimum + eps).
A constant column returns zeros with either method. In particular, a single-row input returns all zeros. Do not replace a zero denominator with 1: adding epsilon is part of the formula even for nonconstant columns.
For a small column [0, 2e-7], z-score standardization gives [-0.5, 0.5],
and min-max scaling gives approximately [0, 0.666667]. This makes the
epsilon convention visible.
In a model-training workflow, learn these statistics from training data and reuse them for validation or test data. This exercise only implements the transformation using statistics from its given input.
Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.
Decide which values share a mean and variance
Many normalization methods subtract a mean and divide by a standard deviation:
$$\hat x=\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}.$$
The small positive epsilon keeps the denominator away from zero. The main
difference between methods is which values are used to compute the mean and
variance.
- Batch normalization usually computes one set of statistics per channel, combining examples and spatial positions during training. Its usual inference behavior uses running statistics collected during training.
- Layer normalization computes statistics over selected feature dimensions within each example. Different examples do not share statistics.
- Group normalization divides channels into groups. Within each example, it computes statistics over each group's channels and spatial positions.
For the feature vector [1, 3], the mean is 2 and population variance is 1.
Subtracting the mean gives [-1, 1]; dividing by sqrt(1 + epsilon) leaves values
close to [-1, 1]. Computing statistics over the batch axis instead would combine
values from different examples and produce a different operation.
RMSNorm rescales without centering
RMSNorm divides by the root mean square, with epsilon added for stability:
$$\hat x=\frac{x}{\sqrt{\operatorname{mean}(x^2)+\epsilon}}.$$
It does not subtract the mean, so its output need not have mean zero. For example,
[1, 1] stays close to [1, 1] rather than becoming [0, 0].
Normalization layers may apply learnable scale and bias after normalizing. Their shapes must broadcast over the intended axes. These extra parameters mean that the final output need not have mean zero or variance one.
Match the formula and the source of the statistics
Check whether epsilon goes inside or outside the square root. Also check whether the variance divides by the number of values (population variance) or one fewer (sample variance). These choices change the answer.
For feature preprocessing, estimate statistics on training data and reuse them on validation and test data. Using test data to estimate them leaks information into training. An individual exercise may instead ask you to use only the input it supplies. Follow that contract and handle constant features as specified.
Your code, saved here
Sign in to access your accepted solutions and saved draft history. You can keep reading the lesson without an account.
Sign in to your accountStuck?
PyTorch reference solution
Sign in to track your attempts toward unlocking the reference solution.