Tracking Issue for `proc_macro_expand`

[mystor]

Feature gate: #![feature(proc_macro_expand)]

This is a tracking issue for the TokenStream::expand_expr function for use in proc-macros. This function allows eagerly expanding TokenStreams containing expressions from within a proc-macro, such that they can be modified, giving proc-macros similar functionality to built-in macros like format_args!, concat! and include!.

Public API

impl TokenStream {
    pub fn expand_expr(&self) -> Result<TokenStream, ExpandError>;
}

#[non_exhaustive]
pub struct ExpandError;

impl Debug for ExpandError { ... }
impl Display for ExpandError { ... }
impl Error for ExpandError {}
impl !Send for ExpandError {}
impl !Sync for ExpandError {}

Steps / History

• Implementation for literal expressions: proc_macro: Add an expand_expr method to TokenStream #87264
• Generalization for other expression types: TODO
• (optional) Generalization to other AST types: TODO
• Final comment period (FCP)
• Stabilization PR

Unresolved Questions

• What types of expressions beyond literals should be supported?
• Should expanding non-expression macros (e.g. types, items, etc.) be supported?
• Should certain macros (e.g. those with #[allow_internal_unstable]) be expandable or left unexpanded?
• How can we evolve which macros are expanded or not within the rustc stability model? Can downstream crates opt-in to additional macro expansion on nightly channels? Should an additional argument be provided to customize what should be expanded?
• Should compilation be allowed to proceed after an expand method errors?
• Are there concerns about stabilizing the span information generated by internal macros like include!?
• How can we properly preserve span information when expanding expression macros?

[Kestrer]

FYI: allowing expand_expr to expand to any arbitrary expression would be unsound, because unsafe macros rely on their output not being tampered with for soundness. Take for example this innocuous-looking proc macro:

#[proc_macro]
pub fn expand_into(stream: TokenStream) -> TokenStream {
    let mut parts = stream.into_iter();
    let Some(TokenTree::Group(unexpanded)) = parts.next() else { panic!() };
    let into = parts.collect::<TokenStream>();
    let expanded = unexpanded.expand_expr().unwrap();
    quote!(#into!(#expanded)).into()
}

By combining it with a safe macro that emits unsafe like pin_utils::pin_mut, one can violate pinning guarantees:

macro_rules! helper {
    ({
        let $output:ident;
        let mut $x1:ident = $x2:ident;
        let mut $x3:ident = $e:expr;
    }) => {
        let mut $output = $e;
    };
}

let future = async {
    let value = 5;
    let reference = &value;
    yield_now().await;
    println!("{reference}");
};

expand_into!([{ let pinned; pin_mut!(future) }] helper);
// expands to: let pinned = unsafe { Pin::new_unchecked(&mut future) };
let _ = pinned.poll(cx);

let moved_future = future;
pin_mut!(moved_future);
moved_future.poll(cx); // oops!

So if the functionality of expand_expr were to be extended to any arbitrary expression, we would have to emit an error if the expansion emits any unsafe blocks. The method can also be supplemented with an expand_expr_unsafe if the user guarantees they won't change the macro's output in an unsound way. Alternatively, macros could somehow mark themselves as safe or unsafe to eagerly expand.

[danielhenrymantilla]

Or a simpler albeit more conceptual example:

//! crate a
#[macro_export]
macro_rules! fancy_block {() => (
    unsafe {} // needless but harmless unsafe: SOUND
)}

//! dependent, which uses no `unsafe` code:

fn main() {
    match! ::a::fancy_block!() {
        ( $some_ident:ident {} ) => (
            $some_ident {
                ::core::hint::unreachable_unchecked()
            }
        );
    }
}

• (where I am using match! as the shorthand for eager expansion, but feel free to imagine it being a combination of expand_into + a helper should match! not be clear).