Skip to content
← All PyTorch tracks

Attention Variants

PyTorch

After Attention 101 — the real-world variants that actually run in modern LLMs.

14 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

  1. 1 Not solved yet. Cross Attention Practice · Medium
  2. 2 Not solved yet. Causal Attention Mask Practice · Easy
  3. 3 Not solved yet. Grouped-Query Attention Practice · Hard
  4. 4 Not solved yet. Sliding Window Attention Practice · Medium
  5. 5 Not solved yet. Causal Attention with Framework Operations Practice · Hard
  6. 6 Not solved yet. KV Cache for Autoregressive Decoding Practice · Hard
  7. 7 Not solved yet. Attention with Online Softmax Practice · Hard
  8. 8 Not solved yet. Causal Self-Attention Block Practice · Medium
  9. 9 Not solved yet. Cross-Attention Block Practice · Medium
  10. 10 Not solved yet. Causal RoPE and SwiGLU Block Practice · Hard
  11. 11 Not solved yet. Encoder-Decoder Transformer Forward Pass Practice · Hard
  12. 12 Not solved yet. Train Encoder-Decoder Seq2Seq Step Practice · Hard
  13. 13 Not solved yet. Encoder-Decoder Greedy Decode Practice · Medium
  14. 14 Not solved yet. Encoder-Decoder Beam Search Practice · Hard

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

With the query-head count fixed, what tradeoff does grouped-query attention offer between multi-head and multi-query attention?

Choose one answer to reveal the explanations. Answers are kept only while this page is open. 

query_heads = 8
for name, kv_heads in [("MHA", 8), ("GQA", 2), ("MQA", 1)]:
    print(name, "query heads per KV head:", query_heads // kv_heads)
Question 1 of 4