Skip to content

← Position Encodings step 1 of 5

Hard Primitives

Implement Positional Encoding

Sinusoidal positional encoding assigns a fixed vector to each sequence position. Adjacent feature pairs use sine and cosine of the same angle. Later pairs change more slowly as the position increases.

Your task

Implement positional_encoding(seq_len, d_model). Both inputs are positive integers, and d_model is even. Return a floating-point array of shape (seq_len, d_model). This task generates the encoding table only; it does not take token embeddings or add the table to them.

Positions start at zero. For each position pos and pair index i = 0, ..., d_model // 2 - 1, compute:

angle = pos / 10000**(2 * i / d_model)
PE[pos, 2 * i] = sin(angle)
PE[pos, 2 * i + 1] = cos(angle)

Angles are in radians. i identifies a sine/cosine pair, not a column: columns 2*i and 2*i+1 share one frequency. Use ordinary division in the exponent and ** for exponentiation in Python.

Example

For seq_len = 2 and d_model = 4, the two pair frequencies are 1 and 1/100. The output is approximately:

[[0.000000, 1.000000, 0.000000, 1.000000],
 [0.841471, 0.540302, 0.010000, 0.999950]]

Position zero always alternates zero and one. The first pair at position one uses angle 1, while the second uses angle 0.01. This difference in frequency is what makes the columns vary at different rates.

Build a column of positions and a row of pair frequencies, then broadcast their product to form all angles at once. Interleave sine and cosine in the output rather than putting all sine columns before all cosine columns.

The visualization starts with a sinusoidal table. Rows are positions and columns are features; other tabs illustrate different ways to use positions.

Loading visualization…