We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
← Closures and Iterators step 13 of 28
peekable and next_if: write a tokeniser
Turn a string into tokens without losing the character that ends a run.
pub fn tokenize(src: String) -> Vec<String>
Scan from left to right using Rust char values. At each character:
- If
is_whitespace()is true, skip it. - If
is_ascii_digit()is true, collect the longest consecutive run of ASCII digits into one token. - Otherwise, if
is_alphabetic()is true, collect the longest consecutive run of characters matching that predicate into one token. - Emit any other character as a token on its own.
Preserve token text and order. Do not normalize Unicode or change case.
These are character-predicate rules, not the rules of a programming language:
underscores do not join words, signs do not join numbers, and a decimal point
is a separate token. Alphabetic runs use Unicode's alphabetic property;
numeric runs accept only ASCII 0 through 9.
"a1+22b" -> ["a", "1", "+", "22", "b"]
"x = 42;" -> ["x", "=", "42", ";"]
"héllo Ω9" -> ["héllo", "Ω", "9"]
"foo_bar++" -> ["foo", "_", "bar", "+", "+"]
"-12.5" -> ["-", "12", ".", "5"]
"" -> []
Whitespace-only input also returns an empty vector. One char is a Unicode
scalar value, which need not be a whole visible character. Use the stated
predicates rather than trying to split text into visual symbols.
Fix the iterator move first
The starter calls it.take_while(...) while its outer loop still needs it.
take_while takes its receiver by value, so this moves the Chars iterator
into the adapter. The next loop iteration tries to use the moved iterator,
causing E0382.
by_ref() lets an adapter use a mutable reference to an iterator instead.
The original iterator remains available after that borrow ends. This example
takes two items and then reads the remainder:
fn main() {
let mut it = [1, 2, 3, 4].into_iter();
let first: Vec<_> = it.by_ref().take(2).collect();
let rest: Vec<_> = it.collect();
assert_eq!(first, [1, 2]);
assert_eq!(rest, [3, 4]);
}
Do not consume the run's boundary
Adding by_ref() alone does not fix this tokenizer. take_while reads the
first item that fails its predicate and does not return it to the source
iterator. For "12+3", that would lose the +.
fn main() {
let mut it = [1, 2, 3, 4].into_iter();
let first: Vec<_> = it.by_ref().take_while(|&n| n < 3).collect();
let rest: Vec<_> = it.collect();
assert_eq!(first, [1, 2]);
assert_eq!(rest, [4]);
}
The rejected item has been consumed from this iterator. That does not mean any value it might refer to has been destroyed: iterators can yield references as well as owned values.
Keep one Peekable for the whole scan
Create src.chars().peekable() before the loop. peek() lets you inspect the
next item while keeping it available to the wrapper's next operation. It may
advance the underlying iterator to fill its buffer.
next_if(predicate) returns the next item only when the predicate accepts it.
Otherwise it leaves that item available for later use through the same
Peekable. For example:
fn main() {
let mut it = "12+".chars().peekable();
assert_eq!(it.next_if(char::is_ascii_digit), Some('1'));
assert_eq!(it.next_if(char::is_ascii_digit), Some('2'));
assert_eq!(it.next_if(char::is_ascii_digit), None);
assert_eq!(it.next(), Some('+'));
}
Do not create a temporary Peekable inside each run-extension attempt. When
that wrapper is dropped, a buffered boundary character can be dropped with
it even though the underlying iterator has already advanced.
Handle both kinds of run
The starter only attempts digit runs; it also needs an alphabetic-run branch.
After the outer while let Some(c) = it.next() reads the first character,
use repeated next_if calls to extend the matching run. Skip whitespace and
emit other characters directly.
next_if passes a reference to the item to its predicate. For a char
iterator, char::is_ascii_digit accepts that reference directly. For
is_alphabetic, use |d| d.is_alphabetic() so the method call handles the
reference to the Copy character.
A while let loop allows the body to advance the iterator again. A direct
for c in it moves it into the loop; for c in it.by_ref() instead borrows
it for the loop, which still prevents using it mutably again in the body.
A passing submission must compile, pass the tests, and pass
clippy -D warnings.
Your code, saved here
Sign in to access your accepted solutions and saved draft history. You can keep reading the lesson without an account.
Sign in to your accountStuck?
Rust reference solution
Sign in to track your attempts toward unlocking the reference solution.