Easy End-to-End

Gradient Clipping

Norm clipping limits the total magnitude of a gradient update while keeping its direction. Treat every tensor in the input list as part of one long vector, even when the tensors have different shapes.

Your task

Implement clip_gradients(gradients, max_norm). The input is a list of floating-point gradient tensors and a finite, nonnegative norm limit. Compute the original global L2 norm:

global_norm = sqrt(sum of every squared element in every gradient tensor)

If global_norm > max_norm, multiply every tensor by the same factor max_norm / global_norm. Otherwise, leave its values unchanged. Preserve the inputs and the order and shapes of the returned tensors. Do not update model parameters.

Return a dictionary with:

"clipped_gradients": the list of resulting tensors.
"global_norm": the original norm as a Python float, rounded to seven decimal places.

Use the unrounded norm for the comparison and scaling. Round only the reported scalar. Do not report the smaller norm after clipping.

One norm, one scale

For gradients [3] and [4], the global norm is 5. With max_norm = 2.5, scale both by 0.5, returning [1.5] and [2]. Report global_norm = 5.0.

Clipping each tensor separately would give a different result. Clamping individual elements to a range such as [-max_norm, max_norm] is also a different operation. Global norm clipping uses one scale for all elements, including negative ones.

Boundary cases

If the norm equals the limit, no clipping is needed. All-zero gradients have norm zero and remain zero, including when max_norm = 0; avoid a 0 / 0 calculation. A positive norm with max_norm = 0 yields zero gradients. An empty list has norm zero and returns an empty list.

Clipping can limit a large update, but it does not by itself guarantee stable training or fix nonfinite gradients. This exercise uses finite values whose squared sums fit in the calculation dtype.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Set the learning rate and control gradient size

A learning-rate schedule changes the step size during training. Check its exact indexing: a warmup may start at zero or at a first positive fraction. Test the initial value, the warmup boundary, the decay endpoint, and any later steps. The schedule's name alone does not specify those choices.

Global norm clipping scales all gradients by the same factor:

$$\min\left(1,\frac{c}{\|g\|}\right),$$

where g represents all parameter gradients together and c is the norm limit. For scalar gradients 3 and 4, the global norm is 5. Clipping to 2.5 multiplies both by 0.5, giving [1.5, 2]. Clipping each tensor separately can change their relative direction. Handle zero norm without dividing by zero.

Keep a smoothed copy of the parameters

An exponential moving average (EMA) updates a separate parameter copy:

$$\bar\theta_t=\beta\bar\theta_{t-1}+(1-\beta)\theta_t.$$

For old average 2, new parameter 4, and beta = 0.9, the new average is 0.9*2 + 0.1*4 = 2.2. A larger beta retains more of the past. Initialization and whether EMA runs before or after the optimizer step affect the result.

Use validation to decide when to stop

Evaluate using the intended inference behavior without changing model parameters. For mean per-example loss, weight each batch's contribution by its example count. Giving unequal batches equal weight changes the overall mean.

Early stopping monitors a validation metric and stops after a specified number of observations without improvement. Define ties, the minimum improvement, and patience precisely. Save an independent copy of the best parameters: the final step may not be the best one. Keep the test set out of the stopping decision so it remains an independent evaluation.

Detecting runtime…