Roadmap

Curated paths through the problem set. Pick a track and work through it in order.

ML Foundations

Core building blocks. Start here.

10 problems

1 Implement ReLU easy 2 Implement Sigmoid easy 3 Implement Softmax easy 4 Implement Tanh easy 5 Implement Mean Squared Error easy 6 Implement Binary Cross-Entropy Loss medium 7 Implement Cross-Entropy Loss medium 8 Implement L2 Regularization easy 9 Implement Gradient Descent Step easy 10 Implement Cosine Similarity easy

Neural Network Layers

Build the components of a neural network from scratch.

9 problems

1 Implement Linear Layer medium 2 Implement Batch Normalization medium 3 Implement Layer Normalization medium 4 Implement Dropout medium 5 Implement Embedding Lookup easy 6 Implement Max Pooling 1D medium 7 Implement Average Pooling 1D medium 8 Implement 1D Convolution hard 9 Skip Connection Block medium

Optimizers & Training

From SGD to Adam. Understand how models learn.

8 problems

1 Implement Gradient Descent Step easy 2 Implement Momentum Update medium 3 Implement Adam Optimizer Step hard 4 Training Loop easy 5 Mini-Batch Training medium 6 Learning Rate Scheduler medium 7 Gradient Clipping easy 8 Weight Initialization easy

Model Architectures

Build complete models end to end.

9 problems

1 Two-Layer MLP easy 2 Binary Classifier easy 3 Multi-Class Classifier easy 4 Simple CNN medium 5 Simple RNN Cell medium 6 LSTM Cell medium 7 GRU Cell medium 8 Autoencoder medium 9 Simple GAN Generator hard

Attention & Transformers

From dot-product attention to full transformer blocks.

8 problems

1 Implement Scaled Dot-Product Attention hard 2 Implement Positional Encoding hard 3 Self-Attention Layer hard 4 Cross Attention medium 5 Multi-Query Attention medium 6 Grouped-Query Attention hard 7 Transformer Encoder Block hard 8 Transformer Decoder Block hard

Modern Techniques

Cutting-edge methods from recent papers.

10 problems

1 RMSNorm easy 2 GELU Activation easy 3 SwiGLU Activation medium 4 Rotary Position Embeddings hard 5 ALiBi Position Bias medium 6 LoRA Update medium 7 KV Cache for Autoregressive Decoding hard 8 Mixture of Experts Routing hard 9 Nucleus (Top-P) Sampling medium 10 Speculative Decoding Accept/Reject hard