← All PyTorch tracks

Tokenization & Embeddings

PyTorch

From raw text to token tensors — BPE, subword, and the embedding matrices that turn ids into vectors.

15 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

1 Not solved yet. Implement Embedding Lookup Practice · Easy
2 Not solved yet. BPE Merge Step Practice · Medium
3 Not solved yet. BPE Encode Text Practice · Hard
4 Not solved yet. Tokenize and Pad Batch Practice · Medium
5 Not solved yet. Learned Absolute Position Embedding Practice · Easy
6 Not solved yet. Tied Input/Output Embeddings Practice · Medium
7 Not solved yet. Subword Tokenizer: Greedy Longest-Prefix-Match Practice · Medium
8 Not solved yet. MLM Masking Strategy Practice · Medium
9 Not solved yet. MLM Forward Pass Practice · Medium
10 Not solved yet. Train MLM Pretraining Step Practice · Hard
11 Not solved yet. MLM Forward with Tied Output Head Practice · Medium
12 Not solved yet. MLM Evaluation: Masked Accuracy Practice · Medium
13 Not solved yet. Causal LM Forward Pass Practice · Medium
14 Not solved yet. Train Causal LM Pretraining Step Practice · Hard
15 Not solved yet. Train Tiny GPT End-to-End Practice · Hard

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

Starting with no accumulated gradients, which rows have nonzero gradients after this loss is differentiated?

Choose one answer to reveal the explanations. Answers are kept only while this page is open.

import torch
from torch import nn

emb = nn.Embedding(50_000, 256)
loss = emb(torch.tensor([7, 42, 7])).sum()
loss.backward()
grad = emb.weight.grad
print(grad.abs().sum(dim=1).nonzero().flatten().tolist())
print(grad[7, 0].item(), grad[42, 0].item())
print(grad.is_sparse)

Question 1 of 4