Include based generated sources from time-tz fails autocomplete

[Yuri6037]

In projects which uses build-scripts to generate rust source, rust-analyzer fails to provide any autocomplete or type information. Link to an example GitHub project that reproduces the issue: https://github.com/Yuri6037/time-tz.

rust-analyzer version: 0.4.1598-standalone (99718d0c8 2023-07-22)

rustc version: rustc 1.71.0 (8ede3aae2 2023-07-12)

relevant settings: macOS Ventura, rustup install, no specific environment variable.

[flodiebold]

Probably same problem as #15213.

[Yuri6037]

It might be a size limit as I've just checked another of my project which also uses include! macro to include a generated RS file and it works.

Does rust-analyzer limit in any way the size in bytes, characters, types, etc that a generated include! file has?

[lnicola]

Looks like it:

[Yuri6037]

Looks like it:

How did you get this error as I don't have any error that shows up on my side?

[lnicola]

No idea, I just have it if I open that file. But I can confirm that timezones::db::europe::LONDON resolve after bumping that limit.

[Yuri6037]

No idea, I just have it if I open that file. But I can confirm that timezones::db::europe::LONDON resolve after bumping that limit.

I see, so there are possibly 2 bugs:

• The error does not work under macOS systems.
• The token limit is too low.

I think for the second problem the token limit should probably be a configuration parameter with an option to just disable the token limit.

[lnicola]

The error does not work under macOS systems.

Are you sure you don't see it if you open src/timezones.rs and look at the Problems pane (or equivalent)? The portable way to check is:

$ rust-analyzer diagnostics .
processing crate: build_script_build, module: build.rs
processing crate: time_tz, module: src/lib.rs
Diagnostic { code: Ra("inactive-code", WeakWarning), message: "code is inactive due to #[cfg] directives: feature = \"system\" is disabled", range: 4414..4456, severity: WeakWarning, unused: true, experimental: false, fixes: None, main_node: None }
Diagnostic { code: Ra("inactive-code", WeakWarning), message: "code is inactive due to #[cfg] directives: feature = \"posix-tz\" is disabled", range: 4458..4504, severity: WeakWarning, unused: true, experimental: false, fixes: None, main_node: None }
processing crate: time_tz, module: src/binary_search.rs
processing crate: time_tz, module: src/timezone_impl.rs
processing crate: time_tz, module: src/timezones.rs
Diagnostic { code: Ra("macro-error", Error), message: "macro invocation exceeds token limit: produced 1211690 tokens, limit is 1048576", range: 1698..1705, severity: Error, unused: false, experimental: true, fixes: None, main_node: None }
processing crate: time_tz, module: src/interface.rs

diagnostic scan complete

Error: diagnostic error detected

The token limit is too low.

I'm not a fan of making it configurable. Every user of your crate would have to bump it.

[lnicola]

I wonder if the limit actually makes sense for include!. Feels like it doesn't.

[Yuri6037]

The error does not work under macOS systems.

Are you sure you don't see it if you open src/timezones.rs and look at the Problems pane (or equivalent)? The portable way to check is:

$ rust-analyzer diagnostics .
processing crate: build_script_build, module: build.rs
processing crate: time_tz, module: src/lib.rs
Diagnostic { code: Ra("inactive-code", WeakWarning), message: "code is inactive due to #[cfg] directives: feature = \"system\" is disabled", range: 4414..4456, severity: WeakWarning, unused: true, experimental: false, fixes: None, main_node: None }
Diagnostic { code: Ra("inactive-code", WeakWarning), message: "code is inactive due to #[cfg] directives: feature = \"posix-tz\" is disabled", range: 4458..4504, severity: WeakWarning, unused: true, experimental: false, fixes: None, main_node: None }
processing crate: time_tz, module: src/binary_search.rs
processing crate: time_tz, module: src/timezone_impl.rs
processing crate: time_tz, module: src/timezones.rs
Diagnostic { code: Ra("macro-error", Error), message: "macro invocation exceeds token limit: produced 1211690 tokens, limit is 1048576", range: 1698..1705, severity: Error, unused: false, experimental: true, fixes: None, main_node: None }
processing crate: time_tz, module: src/interface.rs

diagnostic scan complete

Error: diagnostic error detected

When I run this command from embedded terminal I get:
error: 'rust-analyzer' is not installed for the toolchain 'stable-aarch64-apple-darwin'

The token limit is too low.

I'm not a fan of making it configurable. Every user of your crate would have to bump it.

I get your point but I'm not sure you'd be willing to increase this limit and even if you do increase this limit there is still the problem that this isn't scalable because you might reach into a project which needs to generate a LOT of tokens and will eventually reach into the same problem again. I'm not sure this is fair to even have such a limit on the number of tokens in rust-analyzer at least if it's lower than rustc itself.

[lnicola]

Yeah, you can run it from the Code extension directory (check the manual for the path) or download it from GitHub.

Personally, I'd agree to bump the limit. I believe not handling correct code is generally a bug. But we can't use the same (set of) limits as rustc because if cargo check hangs, you cancel it. But if the IDE hangs (maybe a macro call expands to itself or whatever), you're going to be very annoyed.

That said, this specific limit doesn't seem useful at all for include! calls.

[Veykril]

Ye we should skip the limit for include! invocations.

I don't think we should make the limit configurable as we do with other things, but instead have the limit be annotatable for the crate or a macro call with a tool attribute like rust_analyzer::macro_expansion_token_limit(10000)