Procedural macros: the three kinds

Every real Rust project uses procedural macros within a day of starting. #[derive(Serialize)], #[tokio::main], #[test], #[async_trait], sqlx::query!. You do not have to write them to be a competent Rust programmer. You absolutely do have to be able to read them and reason about what they do to your code.

Why this is an article and not a problem

Put #[proc_macro] in an ordinary binary and the compiler says:

error: the `#[proc_macro]` attribute is only usable with crates of the
       `proc-macro` crate type

Compile the same file with --crate-type proc-macro and it succeeds — but what comes out is a compiler plugin, not a runnable program. main is dead code in it. There is no arrangement of one file that both defines a proc macro and runs it.

The reason is bootstrapping, and it is worth understanding rather than memorising. A procedural macro is a native dynamic library that rustc loads and executes while compiling the crate that uses it. So it must already be compiled — fully, to machine code — before compilation of the consumer can begin. A crate cannot be its own prerequisite. That is why every proc macro in the ecosystem lives in a separate crate, and why you see paired crates like serde / serde_derive and thiserror / thiserror-impl.

Three consequences follow immediately:

• A proc-macro crate can export only macros. No helper types, no traits, no constants. This is the entire reason for the serde / serde_derive split: the traits have to live somewhere that is not the macro crate.
• A proc-macro crate cannot use its own macros, not even in its own tests. Testing is done from a separate crate, usually with trybuild.
• A proc macro runs arbitrary code at compile time on the developer’s machine. It can read files, open sockets, spawn processes. This is a genuine supply-chain consideration, and it is why “we do not add proc-macro dependencies casually” is a defensible engineering position rather than paranoia.

The three kinds

1. Function-like

#[proc_macro]
pub fn sql(input: TokenStream) -> TokenStream

Invoked as sql!(SELECT * FROM users). Takes the tokens between the delimiters, returns replacement tokens. The same shape as a macro_rules! macro from the caller’s point of view — the difference is what it can do with the input.

The advantage over macro_rules! is that the input need not be valid Rust at all. It only has to lex. sql!(SELECT * FROM users WHERE id = 3) is not a Rust expression and never will be, but it tokenises fine, and a proc macro can parse it however it likes. serde_json::json!({"a": [1, 2]}) is the same trick.

2. Derive

#[proc_macro_derive(Describe, attributes(describe_skip))]
pub fn derive_describe(input: TokenStream) -> TokenStream

Attached with #[derive(Describe)]. The input is the entire annotated item — the whole struct or enum, tokens and all.

The critical detail, and the one that confuses everyone at least once: the output is appended alongside the original item, which is not replaced. A derive macro can look at your struct and add things next to it. It cannot modify it, cannot add a field, cannot change a type, cannot remove the pub. If you have ever wondered why no derive macro in the ecosystem ever alters the struct it is attached to — that is why.

attributes(describe_skip) declares helper attributes: inert attributes that may appear on the item’s fields and are visible to your macro. Without that declaration, #[describe_skip] on a field is an unknown-attribute error. This is how #[serde(rename = "…")] works — serde_derive declares serde as a helper.

3. Attribute

#[proc_macro_attribute]
pub fn instrument(attr: TokenStream, item: TokenStream) -> TokenStream

Attached as #[instrument] or #[instrument(level = "debug")]. Two token streams arrive:

• attr — the macro’s own arguments, level = "debug". Empty if there are none.
• item — the entire annotated item.

And the return value replaces the item entirely. Not appended: replaced. An attribute macro can delete your function and emit something completely different in its place, and several popular ones effectively do.

Derive appends, attribute replaces

This is the single most useful pair of facts in the whole area, so it is worth stating as a table:

	input	output
function-like	the tokens inside !(...)	replaces the invocation
derive	the whole item	appended next to it; item untouched
attribute	args + the whole item	replaces the item

If you are ever unsure whether some annotation could have changed your function’s body, the answer is: #[derive] no, #[attribute] yes.

Reading a proc macro

You are going to open one eventually. The shape is always the same:

use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};

#[proc_macro_derive(Describe)]
pub fn derive_describe(input: TokenStream) -> TokenStream {
    let ast = parse_macro_input!(input as DeriveInput);   // tokens -> syntax tree
    let name = &ast.ident;

    let expanded = quote! {                               // syntax tree -> tokens
        impl Describe for #name {
            fn describe(&self) -> String {
                ::std::string::String::from(stringify!(#name))
            }
        }
    };

    expanded.into()
}

Three crates do essentially all of the work, and you will see them in every proc-macro crate you ever open:

• proc-macro2 — a mirror of the compiler’s TokenStream type that works outside a proc-macro context, so the logic can be unit tested.
• syn — the parser. Turns a TokenStream into a real syntax tree: DeriveInput, ItemFn, Expr, Type, and about two hundred others.
• quote — the printer. quote!{ ... } is a quasi-quotation: you write Rust, and #name splices a value in.

Note that interpolation in quote! is #name, not $name. After a week of macro_rules! this will catch you at least once.

Where this goes

Item 18.18 traces #[derive(Debug)] end to end and shows exactly what #[tokio::main] rewrites your main into. Item 18.19 covers the property that makes generated code look so paranoid: procedural macros have no hygiene at all. Item 18.20 is the decision procedure.