We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
← Tokenization & Embeddings step 2 of 15
BPE Merge Step
Merge a specified pair of adjacent tokens in one left-to-right pass. This is the basic operation used by the simplified BPE encoder in the next exercise. BPE stands for Byte-Pair Encoding; here, tokens are Python strings.
Your task
Implement bpe_merge_step(tokens, pair).
tokensis a list of token strings. A token may contain more than one character.pairis a two-element list of strings to match exactly, in that order.- Return a new list of strings. Leave both input lists unchanged.
Read the original sequence from left to right. When two adjacent tokens match pair, append their concatenation and consume both tokens. Otherwise, append the current token and move forward by one. Do not scan the newly created output again during this step.
For tokens ["l", "o", "w", "e", "r"] and pair ["l", "o"], return
["lo", "w", "e", "r"]. A later step could merge ["lo", "w"], treating
"lo" as one token.
Matches cannot overlap. For ["a", "a", "a"] and pair ["a", "a"],
return ["aa", "a"]: the first two tokens are consumed, leaving the third.
Four a tokens produce ["aa", "aa"], not one "aaaa" token.
Matching is case-sensitive and uses whole tokens, not substrings inside them.
If the pair is absent, return a new list with the original contents. An empty
input returns []; a single token remains unchanged.
This task applies one supplied rule. It does not count pair frequencies, learn merge rules or assign vocabulary IDs. Ordinary Python list and string operations are sufficient in both framework modes.
Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.
Convert text into token IDs
A tokenizer decides which units of text the model sees. Whitespace splitting, greedy subword matching, and byte-pair encoding (BPE) use different rules. They can agree on one string and produce different results on another.
A greedy longest-prefix tokenizer chooses the longest vocabulary entry matching
the current position, emits its ID, then advances. With vocabulary a, ab,
and b, the text ab becomes one token. When no entry matches, handle the
unknown input and advance according to the task's rule. Otherwise the scan can
loop forever. A simplified exercise may omit WordPiece features such as
continuation markers and whole-word unknown handling.
Separate BPE training from encoding
BPE training learns merges by repeatedly combining frequent neighboring pairs. Encoding uses the learned merge order: apply eligible pairs according to their rank. It does not count frequencies afresh to learn a new tokenizer for each input. Follow the specified handling of overlapping pairs within a merge pass.
Pad a batch without treating padding as content
Sequences with lengths two and four can share a length-four tensor by adding two padding entries to the shorter sequence. An attention mask identifies which positions contain real tokens. Left and right padding place those tokens at different indices, which can affect position IDs and generation logic.
Keep the truncation length, unknown-token ID, and padding ID explicit. If the contract allows the padding ID to appear as real content, token values alone cannot identify padding; retain the lengths or mask. Return the list or tensor representation requested by the exercise.
Your code, saved here
Sign in to access your accepted solutions and saved draft history. You can keep reading the lesson without an account.
Sign in to your accountStuck?
PyTorch reference solution
Sign in to track your attempts toward unlocking the reference solution.