AKA, @matklad have been misunderstanding how macro expansion works this whole time.

Background: originally, I thought about macro expansion process as transforming a stream of tokens into a different strem of tokens:

macro_rules! id {
    (($id:tt)*) => {($id)*}
}

fn main() {
  let foo = 92;
  id!(foo)
}

Here, I thought that token foo gets translated from macro call site to macro expansion site.

This motivated the TokenMap and related abstractions. The idea is that we assign ids to tokens (=tokens have identity), and track those ids through macro expansion. Yesterday, having looked at https://doc.rust-lang.org/stable/proc_macro/struct.Span.html, I concluded that this is not, in fact, how the world works.

Consider these two procedural macros:

#[proc_macro]
pub fn id(args: TokenStream) -> TokenStream {
    args
}

#[proc_macro]
pub fn id2(args: TokenStream) -> TokenStream {
    clone_stream(args)
}

fn clone_stream(ts: TokenStream) -> TokenStream {
    ts.into_iter().map(clone_tree).collect()
}

fn clone_tree(t: TokenTree) -> TokenTree {
    match t {
        TokenTree::Group(orig) => {
            let mut new = Group::new(orig.delimiter(), clone_stream(orig.stream()));
            new.set_span(orig.span());
            TokenTree::Group(new)
        }
        TokenTree::Ident(orig) => TokenTree::Ident(Ident::new(&orig.to_string(), orig.span())),
        TokenTree::Punct(orig) => {
            let mut new = Punct::new(orig.as_char(), orig.spacing());
            new.set_span(orig.span());
            TokenTree::Punct(new)
        }
        TokenTree::Literal(orig) =>  { ... },
    }
}

I believe their semantics is the same -- from rustc point of view, they produce equivalent outputs. The implementation of id2 completely erases identity though.

So, bad news, we need to rewrite TokenMap-based stuff to use something else (and I don't know what that something else would be). Good news -- I think this should make more weird cases like include work in a more out-of-the-box way perhaps?

cc @jonas-schievink , @edwin0cheng