Skip to content

← Tokenization & Embeddings step 3 of 15

Hard End-to-End

BPE Encode Text

Encode text with a simplified, character-based BPE procedure. The merge rules and vocabulary are already supplied; you do not train a tokenizer.

Your task

Implement bpe_encode_text(text, vocab, merges) and return a Python list of integer token IDs. Leave vocab and merges unchanged.

  • text is a Python string.
  • merges is an ordered list of two-string rules.
  • vocab maps token strings to integer IDs. It contains every token left after all merge passes. It need not contain characters that are completely merged away.

Follow these steps exactly:

  1. Start with list(text), one token per Python character.
  2. Visit each rule once in the supplied order. For that rule, scan the current tokens left to right and merge matching adjacent pairs without overlap. Use the resulting sequence as the input to the next rule.
  3. After the last rule, look up each remaining token in vocab and return its ID.

For text "lower" and rules [["l", "o"], ["e", "r"], ["lo", "w"], ["low", "er"]]:

start:   ["l", "o", "w", "e", "r"]
pass 1:  ["lo", "w", "e", "r"]
pass 2:  ["lo", "w", "er"]
pass 3:  ["low", "er"]
pass 4:  ["lower"]

If vocab maps "lower" to 6, return [6]. Intermediate tokens do not need IDs because vocabulary lookup happens only after all passes.

Order and text handling

A rule that finds no match leaves the sequence unchanged. Do not restart the rule list when a later merge creates a token, and do not reorder rules by frequency or token length. Each pass has the same behavior as the BPE Merge Step exercise: newly merged output is not rescanned within that pass.

Treat spaces, punctuation and case literally. Do not split into words, strip whitespace, normalize characters or convert the string to UTF-8 bytes. Python characters are Unicode code points, which need not match visually perceived characters. An empty string returns []; with no rules, look up the original characters directly.

This exercise defines a small ordered-pass encoder. It does not reproduce a particular model's production tokenizer, which may use byte encoding, pre-tokenization and special tokens. Vocabulary IDs are supplied labels, not positions you assign from token order or length.