Skip to content
← All tracks

Position Encodings

PyTorch

Sinusoidal, learned, relative, RoPE, ALiBi — how transformers know where tokens live.

0 / 5 solved
Continue →
  1. 1. Not solved yet. Implement Positional Encoding
  2. 2. Not solved yet. Relative Position Encoding
  3. 3. Not solved yet. Rotary Position Embeddings
  4. 4. Not solved yet. ALiBi Position Bias
  5. 5. Not solved yet. Transformer Block with RoPE

Check yourself

4 questions · one attempt each

These do not count toward finishing the track. They are here to catch the things that are easy to read past.

0 / 4

What does a relative position scheme give that an absolute one does not? 

# absolute: x + PE(pos)
# relative: attention score depends on (i - j)
Question 1 of 4