Subword Tokenizer: Greedy Longest-Prefix-Match

Split a string into vocabulary entries by repeatedly choosing the longest match at the current position. This is a simplified greedy subword tokenizer: it uses literal strings, without word splitting, continuation markers, normalization or unknown-token handling.

Your task

Implement subword_tokenize(text, vocab) with ordinary Python string and list operations. Return a list[str] in text order, not token IDs or a tensor.

• text is the string to split. Preserve its exact contents, including case, whitespace and punctuation.
• vocab is a list of token strings. It contains every individual character that appears in text, so at least one nonempty match always exists.

Starting at position 0:

1. Find every vocabulary entry that matches the start of the remaining text.
2. Choose the longest nonempty match and append it to the output.
3. Advance by its length and repeat until no text remains.

Vocabulary order and duplicate entries do not affect the result. Empty vocabulary entries, if present, are never emitted. An empty input string returns []. Joining the returned tokens must reproduce text exactly. Leave the supplied vocabulary list unchanged.

For text="cats" and vocab=["c", "a", "t", "s", "ca", "cat"], choose "cat" first, then "s", giving ["cat", "s"].

Greedy does not mean fewest tokens

Do not backtrack or search for a globally shortest tokenization. For text="abcdef" and a vocabulary containing each character plus "abc", "abcd" and "def", return ["abcd", "e", "f"]. Choosing "abc" then "def" would use fewer tokens, but would break the longest-match rule at the first position.

To implement the search, try endpoints from the end of the text down toward the current position and stop at the first vocabulary match. A set makes membership checks convenient, but substring creation and the search still have a cost. This exercise does not implement vocabulary training or a full WordPiece tokenizer.