Skip to content

← Closures and Iterators step 13 of 28

Hard End-to-End

peekable and next_if: write a tokeniser

Turn a string into tokens without losing the character that ends a run.

pub fn tokenize(src: String) -> Vec<String>

Scan from left to right using Rust char values. At each character:

  1. If is_whitespace() is true, skip it.
  2. If is_ascii_digit() is true, collect the longest consecutive run of ASCII digits into one token.
  3. Otherwise, if is_alphabetic() is true, collect the longest consecutive run of characters matching that predicate into one token.
  4. Emit any other character as a token on its own.

Preserve token text and order. Do not normalize Unicode or change case. These are character-predicate rules, not the rules of a programming language: underscores do not join words, signs do not join numbers, and a decimal point is a separate token. Alphabetic runs use Unicode's alphabetic property; numeric runs accept only ASCII 0 through 9.

"a1+22b"      -> ["a", "1", "+", "22", "b"]
"x = 42;"     -> ["x", "=", "42", ";"]
"héllo Ω9"    -> ["héllo", "Ω", "9"]
"foo_bar++"   -> ["foo", "_", "bar", "+", "+"]
"-12.5"       -> ["-", "12", ".", "5"]
""            -> []

Whitespace-only input also returns an empty vector. One char is a Unicode scalar value, which need not be a whole visible character. Use the stated predicates rather than trying to split text into visual symbols.

Fix the iterator move first

The starter calls it.take_while(...) while its outer loop still needs it. take_while takes its receiver by value, so this moves the Chars iterator into the adapter. The next loop iteration tries to use the moved iterator, causing E0382.

by_ref() lets an adapter use a mutable reference to an iterator instead. The original iterator remains available after that borrow ends. This example takes two items and then reads the remainder:

fn main() {
    let mut it = [1, 2, 3, 4].into_iter();
    let first: Vec<_> = it.by_ref().take(2).collect();
    let rest: Vec<_> = it.collect();
    assert_eq!(first, [1, 2]);
    assert_eq!(rest, [3, 4]);
}

Do not consume the run's boundary

Adding by_ref() alone does not fix this tokenizer. take_while reads the first item that fails its predicate and does not return it to the source iterator. For "12+3", that would lose the +.

fn main() {
    let mut it = [1, 2, 3, 4].into_iter();
    let first: Vec<_> = it.by_ref().take_while(|&n| n < 3).collect();
    let rest: Vec<_> = it.collect();
    assert_eq!(first, [1, 2]);
    assert_eq!(rest, [4]);
}

The rejected item has been consumed from this iterator. That does not mean any value it might refer to has been destroyed: iterators can yield references as well as owned values.

Keep one Peekable for the whole scan

Create src.chars().peekable() before the loop. peek() lets you inspect the next item while keeping it available to the wrapper's next operation. It may advance the underlying iterator to fill its buffer.

next_if(predicate) returns the next item only when the predicate accepts it. Otherwise it leaves that item available for later use through the same Peekable. For example:

fn main() {
    let mut it = "12+".chars().peekable();
    assert_eq!(it.next_if(char::is_ascii_digit), Some('1'));
    assert_eq!(it.next_if(char::is_ascii_digit), Some('2'));
    assert_eq!(it.next_if(char::is_ascii_digit), None);
    assert_eq!(it.next(), Some('+'));
}

Do not create a temporary Peekable inside each run-extension attempt. When that wrapper is dropped, a buffered boundary character can be dropped with it even though the underlying iterator has already advanced.

Handle both kinds of run

The starter only attempts digit runs; it also needs an alphabetic-run branch. After the outer while let Some(c) = it.next() reads the first character, use repeated next_if calls to extend the matching run. Skip whitespace and emit other characters directly.

next_if passes a reference to the item to its predicate. For a char iterator, char::is_ascii_digit accepts that reference directly. For is_alphabetic, use |d| d.is_alphabetic() so the method call handles the reference to the Copy character.

A while let loop allows the body to advance the iterator again. A direct for c in it moves it into the loop; for c in it.by_ref() instead borrows it for the loop, which still prevents using it mutably again in the body.

A passing submission must compile, pass the tests, and pass clippy -D warnings.