Production ML — Training Stack

PyTorch

From minimal training loops to gradient accumulation, EMAs, distributed primitives, and checkpoints. Train models at scale.

12 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

1 Not solved yet. Mini-Batch Training Practice · Medium
2 Not solved yet. Training Loop Practice · Easy
3 Not solved yet. Gradient Accumulation Practice · Medium
4 Not solved yet. Gradient Clipping Practice · Easy
5 Not solved yet. Eval Loop with Metrics Practice · Medium
6 Not solved yet. Exponential Moving Average Practice · Medium
7 Not solved yet. Model Checkpointing Practice · Easy
8 Not solved yet. Data Collator with Padding Practice · Medium
9 Not solved yet. Ring All-Reduce Practice · Hard
10 Not solved yet. Distributed Training Step End-to-End Practice · Hard
11 Not solved yet. JIT Compile a Function Practice · Medium
12 Not solved yet. Vectorize with vmap Practice · Medium

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

Why can maintaining authoritative weights in float32 help when some training computations use float16?

Choose one answer to reveal the explanations. Answers are kept only while this page is open.

import torch

for dtype in [torch.float16, torch.float32]:
    weight = torch.tensor(1.0, dtype=dtype)
    updated = weight + torch.tensor(1e-4, dtype=dtype)
    print(dtype, updated.item())

Question 1 of 4