Tokenize and Pad Batch

Convert a list of strings into equally long lists of character IDs. Keep the start of each text if it is too long, and add padding on the right if it is shorter than the requested length.

Your task

Implement tokenize_and_pad_batch(texts, vocab, max_len, pad_id=-1) using ordinary Python strings, dictionaries and lists. Return list[list[int]], not a PyTorch tensor or JAX array.

• texts is a list of strings. Preserve its order.
• vocab maps individual characters to integer IDs. It contains every character in every input text, including characters beyond the cutoff.
• max_len is a nonnegative integer specifying the length of every output row.
• pad_id is the integer to append for padding; its default is -1.

For each string, look up each character's ID, keep the first max_len IDs, and append enough copies of pad_id to reach exactly max_len entries. Use Python string iteration: spaces, punctuation and case matter. Do not split into words, lowercase text, normalize Unicode or insert special tokens.

For texts=["ab", "a", "baba"], vocab={"a": 2, "b": 5}, max_len=3 and pad_id=-1, return:

[[2, 5, -1], [2, -1, -1], [5, 2, 5]]

An empty string produces a row of padding. An empty batch returns []. If max_len=0, return one empty list per input text. Create separate output rows and leave the input list and vocabulary unchanged.

Padding and model inputs

This function only builds ID lists. It does not return an attention mask. Comparing an ID with pad_id identifies padding only when that ID is reserved for padding and never represents a real character. If a real character has the same ID, keep it; the function does not remove matching values.

The default -1 is a sentinel for this exercise, not a safe vocabulary index to pass directly to an embedding lookup. A downstream model needs a valid padding-token ID and suitable masking. Padding to a fixed length is the batching convention used here, not a requirement for every model.