← All PyTorch tracks

Attention 101

PyTorch

From dot products to multi-head transformers. Each step composes onto the next.

5 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

1 Not solved yet. Implement Scaled Dot-Product Attention Practice · Hard
2 Not solved yet. Self-Attention Layer Practice · Hard
3 Not solved yet. Multi-Query Attention Practice · Medium
4 Not solved yet. Transformer Encoder Block Practice · Hard
5 Not solved yet. Multi-Head Attention Block Practice · Hard

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

Why can omitting the 1/sqrt(d_k) scaling make attention harder to train? The two-key example below illustrates the effect for d_k = 128.

Choose one answer to reveal the explanations. Answers are kept only while this page is open.

import math

scores = [6.0, 0.0]  # illustrative scores, not a random-row measurement

def largest_weight(scale):
    weights = [math.exp((s - max(scores)) / scale) for s in scores]
    return max(weights) / sum(weights)

print(round(largest_weight(1), 3))
print(round(largest_weight(math.sqrt(128)), 3))

Question 1 of 4