Skip to content
← All PyTorch tracks

Causal Attention Lab

PyTorch

Build a causal attention pipeline and inspect which earlier tokens each position uses.

Build · inspect · experiment

What you’ll build

  1. Project a sequence into queries, keys and values.
  2. Apply a causal mask and stable row-wise softmax.
  3. Inspect the attention map and test future-token leakage.

Prerequisites: Matrix multiplication, stable softmax and attention shapes. CPU-friendly; a GPU is optional.

Try the other framework →
0 / 1 solved
Start →
  1. 1. Not solved yet. Causal Attention Lab