← All PyTorch tracks

Normalization

PyTorch

BatchNorm, LayerNorm, GroupNorm, RMSNorm, dropout — when to use which, and how each shifts gradients.

6 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

1 Not solved yet. Implement Batch Normalization Practice · Medium
2 Not solved yet. Implement Layer Normalization Practice · Medium
3 Not solved yet. Implement Group Norm Practice · Medium
4 Not solved yet. RMSNorm Practice · Easy
5 Not solved yet. Implement Dropout Practice · Medium
6 Not solved yet. Feature Normalization Pipeline Practice · Easy

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

For this two-dimensional input, which values does each layer use to calculate its normalization statistics?

Choose one answer to reveal the explanations. Answers are kept only while this page is open.

import torch
from torch import nn

t = torch.arange(20, dtype=torch.float32).reshape(4, 5)
layer = nn.LayerNorm(5)(t)
batch = nn.BatchNorm1d(5)(t)
print(torch.allclose(layer.mean(dim=1), torch.zeros(4), atol=1e-6))
print(torch.allclose(batch.mean(dim=0), torch.zeros(5), atol=1e-6))

Question 1 of 4