peekable and next_if: write a tokeniser

Turn a string into tokens.

pub fn tokenize(src: String) -> Vec<String>

Scan left to right, one char at a time:

• a run of one or more characters where char::is_ascii_digit holds is one token;
• a run of one or more characters where char::is_alphabetic holds is one token (Unicode, so "héllo" is a single word);
• any other non-whitespace character is a token on its own — "++" is two tokens, not one;
• whitespace separates and never appears in the output.

"a1+22b"   -> ["a", "1", "+", "22", "b"]
"x = 42;"  -> ["x", "=", "42", ";"]
""         -> []

Why this problem is the hinge of the track

Everything so far has been a pipeline: a fixed chain of adapters applied uniformly to every element. A tokeniser is not that. Whether the next character belongs to the current token depends on what it is, and you cannot find out without looking — and if looking consumes it, you have destroyed the input.

That is the whole difficulty, and Peekable is the whole answer.

The starter shows you the trap first

It tries to grab the rest of a digit run with take_while:

let rest: String = it.take_while(char::is_ascii_digit).collect();

error[E0382]: borrow of moved value: `it`

Every adapter takes self by value. take_while(self, ..), map(self, ..), filter(self, ..) — all of them consume the iterator and return a new one that owns it. So that line moved it into a TakeWhile, and the next loop iteration has nothing left to call next on.

This error is worth recognising on sight, because it appears every time you try to use an adapter on an iterator you intend to keep using.

by_ref — the fix that is not the answer here

Iterator::by_ref returns &mut Self, and &mut I is itself an Iterator whenever I is. So you can hand the reference to an adapter, and keep the iterator:

let v = vec![1, 2, 3, 4, 5];
let mut it = v.into_iter();
let first: Vec<i32> = it.by_ref().take(2).collect();
let rest:  Vec<i32> = it.collect();
// first = [1, 2]   rest = [3, 4, 5]

Now try the same thing with take_while:

let a: Vec<i32> = it2.by_ref().take_while(|&x| x < 3).collect();
let b: Vec<i32> = it2.collect();
// a = [1, 2]   b = [4, 5]        <- where did 3 go?

3 is gone. take_while had to pull 3 out of the source in order to test it, the test failed, and there is nowhere to put it back. The item that ends a take_while is always destroyed.

For a tokeniser that is fatal: the character that ends one token is the character that starts the next one. You cannot afford to lose it.

Peekable and next_if

Peekable wraps an iterator and adds a one-item buffer:

fn peek(&mut self) -> Option<&I::Item>
fn next_if(&mut self, f: impl FnOnce(&I::Item) -> bool) -> Option<I::Item>
fn next_if_eq<T>(&mut self, expected: &T) -> Option<I::Item>

peek pulls the next item into the buffer and lends it to you; the item is still there for the following next(). next_if is the one you want here: it peeks, applies your predicate, and consumes the item only if the predicate said yes. That is precisely “extend the run while the characters keep matching”, with no lost item and no lookahead bookkeeping.

while let Some(d) = it.next_if(char::is_ascii_digit) {
    tok.push(d);
}

Note the closure receives &Item, so a method taking &self — like char::is_ascii_digit — can be passed directly with no closure at all. char::is_alphabetic takes self, so it needs |d| d.is_alphabetic(); char is Copy, and the auto-deref makes that work.

Create the Peekable once

The failure mode to watch for, and the one no lint will catch for you:

// WRONG: a fresh Peekable every iteration
while let Some(d) = it.by_ref().peekable().next_if(..) { .. }

Each .peekable() builds a new buffer. Anything the previous one had peeked but not consumed is dropped with it, so characters vanish unpredictably. Build the Peekable once, outside the loop, and keep it.

There is a nursery lint, unused_peekable, that fires when a Peekable is created and never peeked — the usual symptom of this mistake — but it is allow-by-default and will not run here. This one is graded on behaviour.

Structuring the loop

while let Some(c) = it.next() at the top, then branch on what c is. You need it inside the loop body, so a for loop will not do — a for moves the iterator into itself and you would be back at E0382. Clippy’s while_let_on_iterator normally rewrites while let ... = it.next() into a for, but it is smart enough not to fire when the body still uses the iterator, which is exactly this case.

Grade is compile + tests + clippy -D warnings.