PyTorch roadmap

Build your skills one exercise at a time. Choose a sequence and follow it in order, or return to your next unsolved problem. Prefer lessons alongside your practice? Explore the guided tracks.

6 sequences · scroll to explore

ML Foundations

Core building blocks. Start here.

10 problems

Start sequence

Up next: Implement ReLU

1 Implement ReLU Easy 2 Implement Sigmoid Easy 3 Implement Softmax Easy 4 Implement Tanh Easy 5 Implement Mean Squared Error Easy 6 Implement Binary Cross-Entropy Loss Medium 7 Implement Cross-Entropy Loss Medium 8 Implement L2 Regularization Easy 9 Implement Gradient Descent Step Easy 10 Implement Cosine Similarity Easy

Neural Network Layers

Build the components of a neural network from scratch.

9 problems

Start sequence

Up next: Implement Linear Layer

1 Implement Linear Layer Medium 2 Implement Batch Normalization Medium 3 Implement Layer Normalization Medium 4 Implement Dropout Medium 5 Implement Embedding Lookup Easy 6 Implement Max Pooling 1D Medium 7 Implement Average Pooling 1D Medium 8 Implement 1D Convolution Hard 9 Skip Connection Block Medium

Optimizers & Training

From SGD to Adam. Understand how models learn.

8 problems

Start sequence

Up next: Implement Gradient Descent Step

1 Implement Gradient Descent Step Easy 2 Implement Momentum Update Medium 3 Implement Adam Optimizer Step Hard 4 Training Loop Easy 5 Mini-Batch Training Medium 6 Learning Rate Scheduler Medium 7 Gradient Clipping Easy 8 Weight Initialization Easy

Model Architectures

Build complete models end to end.

9 problems

Start sequence

Up next: Two-Layer MLP

1 Two-Layer MLP Easy 2 Binary Classifier Easy 3 Multi-Class Classifier Easy 4 Simple CNN Medium 5 RNN Sequence Forward Pass Medium 6 LSTM Sequence Forward Pass Medium 7 GRU Sequence Forward Pass Medium 8 Autoencoder Medium 9 Simple GAN Generator Hard

Attention & Transformers

From dot-product attention to full transformer blocks.

8 problems

Start sequence

Up next: Implement Scaled Dot-Product Attention

1 Implement Scaled Dot-Product Attention Hard 2 Implement Positional Encoding Hard 3 Self-Attention Layer Hard 4 Cross Attention Medium 5 Multi-Query Attention Medium 6 Grouped-Query Attention Hard 7 Transformer Encoder Block Hard 8 Transformer Decoder Block Hard

Modern Techniques

Cutting-edge methods from recent papers.

10 problems

Start sequence

Up next: RMSNorm

1 RMSNorm Easy 2 GELU Activation Easy 3 SwiGLU Activation Medium 4 Rotary Position Embeddings Hard 5 ALiBi-Style Distance Bias Medium 6 LoRA Update Medium 7 KV Cache for Autoregressive Decoding Hard 8 Mixture of Experts Routing Hard 9 Nucleus (Top-P) Filtering Medium 10 Speculative Decoding Accept/Reject Hard