PyTorch explainers

PyTorch

Long-form theory: intuitions, derivations, and modern variants. Each article has questions sprinkled throughout. Click to reveal the answer when you've thought about it.

All About Attention

A modern theory guide to attention — soft lookup, scaled dot-product, multi-head, induction heads, RoPE and YaRN context extension, GQA/MQA/MLA, sliding window with sinks, paged KV cache, sparse, linear, FlashAttention's online softmax, and the recent generation of small but consequential tweaks.