panic: "compiler/fallback mismatch" when using proc_macro2::fallback::force() escape hatch

[sam0x17]

So I am in a scenario with my docify crate where I am inside a proc macro (so fallback is not used), but I need to access span().source_text() on a syn item that was loaded via IO and parsed with syn::parse_str() directly (i.e., it is an item that has nothing to do with the current proc_macro environment, so fallback should be fine).

Normally I get a regurgitation of the call site when I try to access source_text() under these conditions, but I found your escape hatch via #220 and tried that @dtolnay. That indeed does work as I get the exact source code that I'm looking for from the involved spans.

My code for this is essentially:

proc_macro2::fallback::force();
let span = item.span();
let Some(excerpt) = span.source_text() else {
    return Err(Error::new(
        Span::call_site(),
        "The referenced sample caused span.source_text() to return `None`. \n\
        You should submit a bug report containing the example that caused this."
    ))
};
proc_macro2::fallback::unforce();

Where item is the result of a visitor pattern traversal over a source_file that was created using syn::parse_str().

The problem is in every case I then get a panic at some point in macro expansion:

error: proc macro panicked
  --> examples/integration.rs:47:1
   |
47 | docify::compile_markdown!("examples/markdown_source", "examples/markdown_bin");
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: message: compiler/fallback mismatch

This seems to happen if I have called proc_macro2::fallback::force() anywhere that encloses my access to the span, so for example:

This compiles fine, but of course excerpt ends up being the macro call site instead of anything to do with the file that is loaded, because fallback is not being used. Notably I don't get the panic here (but this also doesn't work):

proc_macro2::fallback::force();
proc_macro2::fallback::unforce();
let span = item.span();
let Some(excerpt) = span.source_text() else {
    return Err(Error::new(
        Span::call_site(),
        "The referenced sample caused span.source_text() to return `None`. \n\
        You should submit a bug report containing the example that caused this."
    ))
};

This gets the proper source_text() for all of my test invocations and I'm able to print them at compile-time and they are perfect, but I get that panic somewhere later in macro expansion for every single invocation:

proc_macro2::fallback::force();
let span = item.span();
proc_macro2::fallback::unforce();
let Some(excerpt) = span.source_text() else {
    return Err(Error::new(
        Span::call_site(),
        "The referenced sample caused span.source_text() to return `None`. \n\
        You should submit a bug report containing the example that caused this."
    ))
};

I have also tried just doing force() without calling unforce(), as well as doing force() and unforce() at the beginning and end of the proc macro instead of inside of the visitor pattern results loop.

No matter what I do I either get the source_text() I want, and a panic, or source_text() doesn't work at all and I get no panics.

I was going to experiment with commenting out the panic and see if there are any other issues lurking if that panic is disabled, but figured I should report the issue here first.

[sam0x17]

Just circling back on this. Is there any more context information you'd like me to provide @dtolnay?

[dtolnay]

Arbitrarily intermixing fallback tokens and compiler tokens is not supported / not meaningful. Tokens that you create during fallback need to be accessed only during fallback and vice versa. So something like:

proc_macro2::fallback::force();
let source_text = item.span().source_text();
proc_macro2::fallback::unforce();
let Some(excerpt) = source_text else {
    return Err(Error::new(Span::call_site(), "..."));
};