Fragment specifiers: the complete tour

Build an ordered cascade of rules, one per fragment specifier, and let it tell you which specifier claims which piece of syntax.

pub fn classify() -> Vec<String>

classify is written for you: it invokes kind_of! thirteen times and also! eleven times, and returns the tag each invocation produced. Your job is to write kind_of!. The starter has only its catch-all, so every answer is currently "tt".

Choosing the wrong specifier is the number-one reason a macro “mysteriously refuses to match”, and the specifier also decides whether the captured fragment stays welded together as one unit — the deepest trap in the whole system, and the subject of the next item.

The complete list

Every one of these compiles on rustc 1.95:

The four facts worth memorising

ident accepts keywords. kind_of!(pub) answers "ident", not "vis", because pub is a perfectly good identifier token. It does not accept _, which is a reserved token in its own right — you will see this in the results.

literal accepts a leading minus. -7 is one literal fragment, not a unary minus applied to 7. This is a special case in the matcher, and it is why vec![-1, -2] needs no special handling.

vis matches the empty visibility. That is what makes item-generating macros readable: a single matcher $v:vis fn $name:ident accepts both pub fn foo and fn foo, with no $(...)?` wrapper. Hence `also!(vis pub fn)` and `also!(vis fn)` both answer `"vis"`. **`stmt` does not consume the trailing semicolon.** This one costs people an afternoon. `$s:stmt matches let z = 1 and leaves the ; sitting in the input, so a matcher of just ($s:stmt)` fails on `let z = 1;` with "no rules expected this token". The fix is to write the semicolon into the matcher yourself: `($s:stmt ;). also!(bare_stmt let z = 1;) answers "no" for exactly this reason, while kind_of!(let z = 1;) answers "stmt".

A corollary: once you have a stmt fragment, it is already a complete statement. Writing $s;` in the transcriber emits a redundant empty statement and rustc's `redundant_semicolons` lint will say so. Just write `$s.

Overlap, and why order is the whole exercise

Most syntax satisfies several specifiers at once. foo is an ident, a path, a ty, an expr, a pat, a stmt and a tt. Since rules are tried top to bottom and the first match wins, the order you write the rules in is the classification.

Two overlaps are worth predicting before you run the tests:

• [u8; 4] and &str are types but not paths. path really means a path: std::vec::Vec, Option<T>, Self::Item. It will not match a bracketed array type or a reference type.
• _ is not an ident, but it is a ty — _ is the inferred-type syntax, as in let v: Vec<_> = .... also!(ty _) answers "ty" and also!(ident _) answers "no".

The edition wrinkle

Edition 2024 widened expr to also match a top-level _ and a top-level const { ... } block. expr_2021 preserves the old behaviour so that a macro written before the change keeps its exact matching semantics. Hence:

also!(expr21 1 + 2)   // "expr21"
also!(expr21 _)       // "no"   — expr_2021 refuses `_`

while kind_of! with an expr rule would accept _ — you cannot see that here because ty claims _ first.

The thing the results will not show you, so read this

Rule fallthrough is not free. When the matcher reaches a rule and the next thing it needs is a fragment, it hands the remaining tokens to the real parser. If the parser rejects the token straight away — a block must start with {, a lifetime must start with ' — the rule fails cheaply and the next rule is tried. But if the parser starts parsing and then hits an error partway through, that is a hard error for the whole invocation. There is no backtracking to the next rule.

That is why also! exists in this problem. Its rules are tagged — also!(pat …), also!(meta …) — so the matcher never has to guess. Try moving pat or item into the kind_of! cascade and you will get errors like “expected type, found keyword ref“ pointing at a call site that looks perfectly innocent. Tagging a rule with a leading keyword is the standard way real macro libraries dodge this, and it is why so many of them look like my_macro!(@internal …).

::: question Why does the cascade put literal and ident before path, ty and expr, rather than the other way round?

Because the cheap, single-token specifiers can never hard-error, and the parser-invoking ones can.

literal, ident, lifetime and tt are decided by looking at one token. If the token is not a literal, the rule fails and the matcher moves on with no damage. path, ty, expr, pat and item invoke the real parser, and once that parser commits it either succeeds or aborts the whole compilation.

So the ordering is not only “most specific first” for correctness — it is also “least dangerous first” for robustness. Put ty above literal and kind_of!(42) still works (a literal is not a type, and the parser rejects 42 as a type immediately), but put item above literal and kind_of!(42) becomes “expected an item keyword”, because item parsing commits. :::

Also worth knowing

There is one more specifier reserved for the future: attempts to use an unstable fragment kind produce E0658, “use of unstable library feature” / “…are unstable”. You will meet E0658 for real in item 18.7, where the internet will hand you a counting idiom that does not exist on stable.