PyTorch roadmap

Ordered runs of problems, and nothing else: no reading, no progress bar, just the sequence. For a syllabus that interleaves written explainers and can be worked through with friends, see tracks.

ML Foundations

Core building blocks. Start here.

10 problems

1 Implement ReLU Easy 2 Implement Sigmoid Easy 3 Implement Softmax Easy 4 Implement Tanh Easy 5 Implement Mean Squared Error Easy 6 Implement Binary Cross-Entropy Loss Medium 7 Implement Cross-Entropy Loss Medium 8 Implement L2 Regularization Easy 9 Implement Gradient Descent Step Easy 10 Implement Cosine Similarity Easy

Neural Network Layers

Build the components of a neural network from scratch.

9 problems

1 Implement Linear Layer Medium 2 Implement Batch Normalization Medium 3 Implement Layer Normalization Medium 4 Implement Dropout Medium 5 Implement Embedding Lookup Easy 6 Implement Max Pooling 1D Medium 7 Implement Average Pooling 1D Medium 8 Implement 1D Convolution Hard 9 Skip Connection Block Medium

Optimizers & Training

From SGD to Adam. Understand how models learn.

8 problems

1 Implement Gradient Descent Step Easy 2 Implement Momentum Update Medium 3 Implement Adam Optimizer Step Hard 4 Training Loop Easy 5 Mini-Batch Training Medium 6 Learning Rate Scheduler Medium 7 Gradient Clipping Easy 8 Weight Initialization Easy

Model Architectures

Build complete models end to end.

9 problems

1 Two-Layer MLP Easy 2 Binary Classifier Easy 3 Multi-Class Classifier Easy 4 Simple CNN Medium 5 Simple RNN Cell Medium 6 LSTM Cell Medium 7 GRU Cell Medium 8 Autoencoder Medium 9 Simple GAN Generator Hard

Attention & Transformers

From dot-product attention to full transformer blocks.

8 problems

1 Implement Scaled Dot-Product Attention Hard 2 Implement Positional Encoding Hard 3 Self-Attention Layer Hard 4 Cross Attention Medium 5 Multi-Query Attention Medium 6 Grouped-Query Attention Hard 7 Transformer Encoder Block Hard 8 Transformer Decoder Block Hard

Modern Techniques

Cutting-edge methods from recent papers.

10 problems

1 RMSNorm Easy 2 GELU Activation Easy 3 SwiGLU Activation Medium 4 Rotary Position Embeddings Hard 5 ALiBi Position Bias Medium 6 LoRA Update Medium 7 KV Cache for Autoregressive Decoding Hard 8 Mixture of Experts Routing Hard 9 Nucleus (Top-P) Sampling Medium 10 Speculative Decoding Accept/Reject Hard