Skip to content
← All PyTorch tracks

Position Encodings

PyTorch

Sinusoidal, learned, relative, RoPE, ALiBi — how transformers know where tokens live.

5 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

  1. 1 Not solved yet. Implement Positional Encoding Practice · Hard
  2. 2 Not solved yet. Relative Position Encoding Practice · Medium
  3. 3 Not solved yet. Rotary Position Embeddings Practice · Hard
  4. 4 Not solved yet. ALiBi-Style Distance Bias Practice · Medium
  5. 5 Not solved yet. Attention Block with RoPE Practice · Hard

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

What structure does a relative positional contribution based on i - j share across attention pairs?

Choose one answer to reveal the explanations. Answers are kept only while this page is open. 

pairs = [(5, 3), (50, 48)]
for query_position, key_position in pairs:
    distance = query_position - key_position
    print(distance)
Question 1 of 4