Errnous "unexpected token in input"

[leehambley]

rust-analyzer version: fd109fb58 2021-05-10 stable

Given the following piece of test code:

    macro_rules! test_driver {
        ($driver_name:ident, $setup:expr) => {
            concat_idents!(
                test_name = "test_",
                $driver_name,
                "_driver_raises_malformed_error_when_migration_has_both_change_and_up_steps",
                {
                    #[test]
                    fn test_name() -> Result<(), String> {
                        let mut driver = $setup;
                        match driver.apply(&malformed_migration()) {
                            Ok(_) => Err(format!("engine did not report malformed error",)),
                            Err(e) => match e {
                                Error::MalformedMigration => Ok(()),
                                _ => Err(format!("Engine returned {:?}, unexpected", e)),
                            },
                        }?;
                        Ok(())
                    }
                }
            );
        };
    }

    // rust-analyzer bug here (?)
    // https://github.com/rust-analyzer/rust-analyzer/issues/6747
    test_driver!(noop_driver, { NoopDriver {} });
    test_driver!(succeed_or_fail_driver, { SucceedOrFailDriver {} });

This is the first time I wrote a macro, so very possible that I made a mistake, but I don't think so, as the example is more-or-less verbatim from the concat_idents crate docs.

I tried restarting the server, and confirmed that after a few seconds the error returns.

Googling led me to #6747, I don't see any other issues open now with "unexpected token in input" error.

I tried to reproduce minimally, but I cannot reproduce it. My best guess is that because Driver in my original example is a trait, something different happens?

Here is the repro case with simple empty structs, showing a clean vscode build, and the crate version for concat_idents

[edwin0cheng]

MWE:

use concat_idents::concat_idents;

fn main() {
    test_b_test_test();
}

concat_idents!(
    test_name = "test_",
    b,
    "_test_test",
    {
        fn test_name() {}
    }
);  // <- ERROR: unexpected token in input

However, the following code works :

use concat_idents::concat_idents as cat;

fn main() {
    test_b_test_test();
}

cat!(
    test_name = "test_",
    b,
    "_test_test",
    {
        fn test_name() {}
    }
);

So It seem like that we incorrectly resolve the name of concat_idents to builtin one.

[edwin0cheng]

After #8830, the resolution is still incorrect :

[Veykril]

I think we consider the prelude to be always #[macro_use]'d which would mean we can't shadow macros in the prelude I think? Given how path resolution works for us atm https://github.com/rust-analyzer/rust-analyzer/blob/9803a9a148e1c5850d2f649ffbc1bb3575cfa929/crates/hir_def/src/nameres/path_resolution.rs#L373-L384
https://github.com/rust-analyzer/rust-analyzer/blob/9803a9a148e1c5850d2f649ffbc1bb3575cfa929/crates/hir_def/src/nameres/collector.rs#L1028-L1034

[jonas-schievink]

I think we consider the prelude to be always #[macro_use]'d

This part seems correct, the crate whose prelude we import gets an implicit #[macro_use], so that stuff like vec![] is in scope.

Even though these are in legacy macro scope, they can apparently be shadowed by non-legacy macros, which doesn't really match what we do, so I'd say path resolution is to blame. We probably need to look into what rustc does here.

[jonas-schievink]

Hmm, it seems like one issue here is that we eagerly try to resolve and expand legacy macros here: https://github.com/rust-analyzer/rust-analyzer/blob/e8b9f43084d6bfa3a77d50dd7759799d7aaf432c/crates/hir_def/src/nameres/collector.rs#L1892

With proper handling of the macro-use-prelude in place, the macro invocation will resolve to the prelude macro there and get expanded accordingly, but that's wrong, since the use shadows it!

If we don't allow that code path to resolve to macro-use-prelude macros, then the macro's path is rewritten to start with self:::

https://github.com/rust-analyzer/rust-analyzer/blob/e8b9f43084d6bfa3a77d50dd7759799d7aaf432c/crates/hir_def/src/nameres/collector.rs#L1940-L1944

...which then means that it'll never resolve to a prelude macro.

We do this rewriting to ensure that resolution during the fixed-point loop does not try to find legacy-scoped macros (since they are lexically scoped, you shouldn't be able to use one if it is defined later in the file).

This might be an indication that we need to model macro invocations and macro_rules! items as adding an entirely new scope in which new legacy macros may be introduced. This is what rustc does.