← All PyTorch tracks

Optimizers from Scratch

PyTorch

Build the workhorse optimizers byte by byte. Start with vanilla SGD and end at Adam.

4 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

1 Not solved yet. Implement Gradient Descent Step Practice · Easy
2 Not solved yet. Implement Momentum Update Practice · Medium
3 Not solved yet. Implement Adam Optimizer Step Practice · Hard
4 Not solved yet. Train with Adam End-to-End Practice · Medium

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

Why do these SGD updates grow while the learning rate and current gradient stay fixed?

Choose one answer to reveal the explanations. Answers are kept only while this page is open.

import torch

p = torch.nn.Parameter(torch.tensor([1.0]))
opt = torch.optim.SGD([p], lr=0.1, momentum=0.9)
for _ in range(3):
    opt.zero_grad()
    p.sum().backward()
    opt.step()
    print(f"{p.item():.4f}")

Question 1 of 4