Resolving proc-macro ABI breakage for rustc nightly users: one way forward

[fasterthanlime]

I think I've found a way forward for #12579 that requires minimal changes and should make the situation better for a whole lot of folks.

Onbrandedly, I'm first going to provide way too much background.

Background

(Disclaimer: some of those details may be inaccurate - inaccuracies here do not make the whole issue irrelevant)

rust-analyzer (aka "ra") largely leverages its own infrastructure to provide code intelligence: for example, it has its own parser, separate from rustc. It leverages the chalk trait engine, whereas in rustc, chalk is behind an unstable flag, and is not fully integrated (I learned that the hard way).

When it comes to procedural macros (aka "proc macros") however, ra leverages the exact same infrastructure rustc does when actually compiling code. This lets ra expand proc macros, and "index" them, just like regular code ("Go to definition" becomes less than useful, but most everything else works fantastically).

How rust-analyzer expands macros right now

Glossing over many details, essentially:

• The rust-analyzer binary, acting as an LSP server (hereafter referred to as ra-lsp), starts rust-analyzer proc-macro, the "proc macro server" (hereafter referred to as proc-macro-srv)
• ra-lsp and proc-macro-srv (separate processes) communicate using newline-delimited JSON messages over stdin/stdout (the JSON API is defined in the proc-macro-api crate)
• the "expand macro" JSON API request contains the path of a "dylib" (dynamic library / shared object) that should be used to expand the macro.
• that dylib is the result of compiling the proc macro crate with a certain version of rustc (remember, proc macros are "just" code that transforms token streams - they end up being dylibs, TIL!)
• proc-macro-srv opens that dylib, checks the version, and depending on the version it reads (from the .rustc section of the dylib), picks one of its abi_x_y modules to talk to it. It also finds a "registrar" symbol.
  • Note that reading the version and finding the registrar symbol involves parsing ELF/Mach-O/COFF with the object crate. I know, amazing!
• proc-macro-srv then uses three different sets of unstable APIs from rustc/libstd's proc_macro crate to act as a "proc macro server" to the proc macro dylib. It expands the macro as requested, then writes the result as JSON to stdout, which ra-lsp is then able to process further.

And there lies the fragility. The proc-macro-srv <=> some proc-macro crate's compiled dylib interface is unstable by design, and there are no plans to stabilize it. It uses its own serialization, allows plugging your own types (an ability which ra uses, using its own "Span", "TokenStream" types from its tt crate), etc.

In fact, that interface (the "proc macro bridge") was changed multiple times in incompatible ways in the past few weeks alone — to improve performance (remove unneeded stringification/parsing, etc.). More changes can be expected in the future, for additional features.

Here's a diagram to summarize:

(edit: the diagram was slightly wrong)

How a single rust-analyzer binary supports multiple rustc versions right now

Because ra should be usable across multiple rustc versions, but the "proc macro bridge" might change from one version to the next, ra adopted a "multi ABI support" proof of concept approach, merged in July 2021: #9550

Because ra is able to read the "rustc version" from proc macro crates' dylibs (e.g. lib_some_proc_macro.so) after it's been built by cargo check (or `cargo clippy, or... etc), it's able to pick from one of the ABIs it knows about: as of two hours ago, that's 1.58, 1.63, and 1.64 (the older ones were just removed).

However, there's several problems with that approach.

One ra binary vs multiple ABIs: the maintenance cost

The first is that it's a maintenance nightmare. I don't want to speak for @jonas-schievink (who I keep seeing in the Git history for that corner of ra), but I've gone through the same step myself of "upgrading the ABI" and a bunch of changes are needed.

• Sources need to be manually copied from https://github.com/rust-lang/rust/tree/master/library/proc_macro/src to the rust-analyzer repo, in a new abi_x_y directory
• lib.rs is renamed to mod.rs
• All the unstable features need to be removed in some way, stuff like:
  • #[stable()] and #[unstable()] attributes
  • negative impls (!Send and !Sync)
• Some symbols need to be re-exported under different names
• etc.

As I was writing this, a PR You can see landed. Since it's mostly additions, you don't really see all the manual work that goes into adjusting upstream code just so it builds within rust-analyzer.

This is not the main problem I'm trying to solve. In fact, the approach I'm suggesting doesn't immediately remove this work. I just think it's a shame.

One ra binary vs multiple ABIs: the rustc nightly issue

At the time of this writing, r-a has been broken on nightly for roughly a month, cf. #12600.

"Broken" is an oversimplification, let me explain:

• For a while, r-a (any version) only worked with rustc nightly 2022-06-08. For any newer rustc, it would fail to expand proc macros.
• On 2022-07-12, fix: Update 1.63 proc macro ABI to match rustc #12747 was merged, which was thought to address the issue. But it didn't: it added the "rustc 1.63 beta" ABI, which is not what nightly uses (as explained in feat: Support the 1.64 nightly proc macro ABI #12795)
• Because today is monday, today (and for another week), r-a stable is broken for rustc nightlies between 2022-06-08 (which used to work) and 2022-07-18

If my understanding is correct, within 24h, there'll be a new "release preview" version of rust-analyzer, and that one will support rustc nightly 2022-07-18. I'm not sure if it'll support nightly 2022-06-08, because the PR only "adds the 1.64 ABI", and "the 1.63 ABI" as it was there previously didn't seem to work with 2022-06-08.

The problems don't stop there. There's no way to know if a version of r-a is compatible with a version of rustc. Versions of rustc that used to work can get broken by r-a upgrades. (One would have to pin the r-a vscode extension version to some older build, since the r-a binary is bundled with the extension now) — so "just picking another nightly" (already not an option for a lot of users) is not even a fix. Even if a "compatibility table" was maintained, it would have to be revised every week (if you only care about r-a stable) or every day (if you care about r-a nightly), and I'm not aware of any infrastructure in place to do automated testing of r-a against rustc versions.

That's still not it. Desperate, I tried installing rust-analyzer from rustup, since it is a rustup component now, and pointing the rust-analyzer vscode extension to it (via the rust-analyzer.server.path vscode setting). But that doesn't work either, because that rust-analyzer binary is also using the "multi ABI support" scheme, and it fails in exactly the same way.

(Also, when ra's proc-macro-srv becomes incompatible with rustc's proc_macro bridge, IntelliJ Rust breaks too, since they literally pull in the proc-macro-srv crate).

Listing concerns

I've reread #12579 maybe 15 times, because there's a lot of concerns here from various sides and I wanted to make sure that whatever I suggested would address them all.

Here are my concerns:

• AM001 (my first name is "Amos"): I want rust-analyzer to never break for nightly users. That means my work colleagues, rustc contributors (it's possible r-a this doesn't work for other, unrelated reasons), anyone using nightly. There's no technical reason it should ever break.

Here are the concerns I've been able to gather from #12579, from the "rustc project" (rust-lang/rust) side of things:

• RU001: We do not want to stabilize the proc_macro bridge. It is inherently unstable so that it may change in the future, for performance or feature reasons. source
• RU002: Shipping any additional files with the rustc dist component falls under T-infra, likely requires coordination across several teams. source

Here are the concerns I've been able to gather from #12579, from the rust-analyzer side of things:

• RA001: We want rust-analyzer to be backwards- and forwards-compatible over "a few rustc versions"
  • Example: some project is pinned to rustc 1.56, yet 1.62 just came out - a recent r-a should still work, with all the new features added since. source
• RA002: We want rust-analyzer to work regardless of whether the rust-analyzer rustup component is installed. (rust can be installed through other sources than rustup.) source
• RA003: We want rust-analyzer to work in "VSCode workspaces" that may contain multiple Cargo workspaces, each using different rust toolchain versions. source
• RA004: It would be nice if rust-analyzer would build with rustc stable, not requiring nightly features or RUSTC_BOOTSTRAP=1. source

A proposed way forward

I'm suggesting adding a new Abi implementation to proc-macro-srv that uses the sysroot version of proc_macro instead of copying+tuning upstream sources regularly.

I've gone ahead and done that in a fork:

• Internal ABI (using sysroot proc_macro crate) fasterthanlime/rust-analyzer#1

This was significantly less work than the other approach (copying+tuning, cf. this branch) and works flawlessly, today, on rust 2022-07-18.

That implementation will:

• Keep working in case of ABI-breaking but API-compatible changes to the proc_macro bridge
• Stop building in case of API-breaking changes to the proc_macro bridge

If it stops building, "rust-analyzer" will show as "missing" in the rustup component history table. This serves as a signal for users that rust-analyzer cannot be used with that rustc nightly.

Breakages will last at most a week, and should be fixed as part of rust-analyzer's already-existent weekly (-ish?) merge with rust-lang/rust: see these pull requests.

I'm suggesting making that Abi implementation be cfg-gated, and that it be the only Abi built (and selected at runtime) for the "rust-analyzer rustup component".

I'm suggesting the non-rustup build of rust-analyzer not set the relevant cfg feature, which would allow it to keep building under rustc stable, and keep using the "multi ABI" compatibility scheme.

Furthermore, I'm suggesting a rust-analyzer shim be added to rustup, so that one can invoke rust-analyzer +nightly just like one can do rustc +nightly or cargo +nightly - but more importantly, so that the rust-analyzer in $PATH is rustup-aware (picks up rust-toolchain.toml, etc.) and all that's needed to use it is to add the rust-analyzer component to rust-toolchain.toml and set rust-analyzer.server.path to "rust-analyzer").

Let's review concerns one by one:

• AM001: As a user, I am now very happy and relieved. By looking at "rustup component history", I know which rustc versions to pick and which to avoid if I want a working rust-analyzer. I can opt into using the rustup version of rust-analyzer with one change to my rust-toolchain.toml and .vscode/settings.json). rust-analyzer proc macros never ever break for me ever again.
• RU001: This doesn't change - the proc_macro bridge is still unstable. r-a's copypasta approach to "multi ABI compatibility" is still mildly iffy to everyone, but nobody is stabilizing anything.
• RU002: No additional files are shipped under the "rustc" dist component. The existing rust-analyzer component is used, with the existing merge and release process. No coordination is required across team (we just need to ask the rustup team nicely for a rust-analyzer shim) and there's actually a good chance of this happening.
• RA001: vscode-marketplace-distributed RA is still backwards-/forwards-compatible with rustc versions just like it was before. This works "well enough" for stable rustc versions (since there's ample time to upgrade proc-macro-srv before stable is minted) and the majority of ra users don't ever need to learn about this issue.
• RA002: this still works, too. the rust-analyzer binary is still bundled with the vscode extension and is the default.
• RA003: I believe the "VSCode workspace" feature would spin up multiple rust-analyzer instances anyway. Some maybe the bundled version, and some may be from rustup (according to vscode settings) - I don't believe r-a currently has any code specifically to support that, and I think things should not end up any worse or better than they are now.
• RA004: rust-analyzer standalone still builds fine under rustc stable, since all the "sysroot proc_macro" code is cfg'd away, including the unstable opt-ins.

In summary

Using cfg gates, we make rust-analyzer support:

• multiple ABIs (just like now) in the standalone (vscode marketplace) build
• the exact same ABI as rustc in the rustup component build

The former has no changes at all, and retains all the things the r-a team cares about (as per #12579).

The latter means r-a is usable with all rustc nightlies, unless the API breaks, in which case it's the component is missing, which can be checked by consulting the rustup component history.

proc_macro bridge API breakages are resolved during the regular rust-lang/rust-analyzer sync with rust-lang/rust. The code still canonically lives under rust-lang/rust-analyzer (and as a git submodule in rust-lang/rust).

A rustup shim is created for rust-analyzer, which makes opting into "rustup rust-analyzer" as simple as having a rust-toolchain.toml with:

[toolchain]
channel = "nightly-2022-07-18" # or whatever
components = [ "rustfmt", "clippy", "rust-analyzer" ]

And a .vscode/settings.json with:

{
  "rust-analyzer.server.path": "rust-analyzer"
}

Bonus: what if we don't want a shim?

The only real drawback I can see here is that r-a devs are used to installing rust-analyzer to ~/.cargo/bin/. A rustup shim might overwrite that, or just fail to install.

There's several alternatives that would work just as well:

• Provide a different setting: "rust-analyzer.server.fromRustup": true
• Extend rustup run to accept a special toolchain string that means "whatever is right for the current directory" (it currently wants a toolchain string like "nightly" or "stable", which is not what you want if you have a rust-toolchain.toml file), and make sure the rust-analyzer vscode settings allow passing a "shell command" rather than just an "executable path"

The point is to avoid having to provide user-specific absolute paths like these:

{
  "rust-analyzer.server.path": "/home/amos/bearcove/rust/build/x86_64-unknown-linux-gnu/stage1-tools-bin/rust-analyzer"
}

...which would make it impossible to share settings across multiple developers of the same Rust project.

[bjorn3]

The only real drawback I can see here is that r-a devs are used to installing rust-analyzer to ~/.cargo/bin/. A rustup shim might overwrite that, or just fail to install.

For rustfmt (or was it clippy) I believe there is logic to make rustup not install the shim if it is already manually installed as in the past you had to use cargo install rather than rustup component add to get it.

[fasterthanlime]

forgot this bit:

Bonus 2: what about a proc-macro-srv component?

In #12579, it was suggested to ship a proc-macro-srv component, which rust-analyzer could then use. That'd move from a single binary with a proc-macro subcommand, to two binaries: one being rust-analyzer, and the other being proc-macro-srv.

I'd like to keep that as out of scope for that discussion since, "in theory it's nice" but it's also a way of stabilizing (a subset of) the proc macro bridge, it's a new rustup component binary file, etc. etc.

I do think r-a's "proc-macro-api" JSON API should be versioned so it's future-proofed (and rust-analyzer 2023-01-01 can be used with a rustc from six months ago), but it's not a prerequisite for the scheme I'm suggesting here.

[lf-]

Re: the last concern with absolute paths, could we make it resolve stuff with PATH? Or possibly leverage rustup toolchain link?

You can already do like, rustc +dev ...

As a (former, burnout sucks) r-a dev I always used the absolute path in my config for prototyping so I'm not sure if the cargo install thing is true for everyone. I don't like cargo install because it clean builds every time.

[bjorn3]

Re: the last concern with absolute paths, could we make it resolve stuff with PATH?

It already does so, but without rustup shim, there is no rust-analyzer binary in your PATH in case you are getting it from rustup.

Or possibly leverage rustup toolchain link?

rustup toolchain link only adds a symlink from ~/.rustup/toolchain/<toolchain> to the toolchain directory. There is no way to overlay one component on top of an existing local toolchain.

[lf-]

Re: the last concern with absolute paths, could we make it resolve stuff with PATH?

It already does so, but without rustup shim, there is no rust-analyzer binary in your PATH in case you are getting it from rustup.

Or possibly leverage rustup toolchain link?

rustup toolchain link only adds a symlink from ~/.rustup/toolchain/<toolchain> to the toolchain directory. There is no way to overlay one component on top of an existing local toolchain.

Could you configure the language server in your editor to be rust-analyzer +dev and have it hit some project toolchain or something else with no changes? Or is there some further fun there?

[bjorn3]

That works provided that the dev toolchain has rust-analyzer added to it. I'm not sure how to convince ./x.py to add tools to the sysroot it created without using ./x.py install or something like that.

[flodiebold]

I think this would be an ok temporary solution, but does not solve all the concerns that making the proc macro server a separate binary solves.

In particular I think you misunderstood RA0003 a bit. I guess I shouldn't have used the term "VSCode workspace" -- just opening a folder containing multiple Cargo projects is enough. rust-analyzer can handle a workspace containing multiple separate crate graphs using different rustc versions just fine, but if the rust toolchain is being overridden in one of the subfolders your approach would not work.

The main problem with this approach in my opinion is that it requires special configuration from the user if they want nightly support to work, though. (And that it fixes the used rust-analyzer version to the nightly version if you want nightly support.)

I think the rust-analyzer shim is being added, btw.

[lnicola]

Breakages will last at most a week, and should be fixed as part of rust-analyzer's already-existent weekly (-ish?) merge with rust-lang/rust: see these pull requests.

One remark about this: what's being merged upstream is the stable release, so if any upstream changes make it fail to build, it will be fixed at the earliest one week later (in the next stable version) + 2 days or so to get merged.

[fasterthanlime]

In particular I think you misunderstood RA0003 a bit. I guess I shouldn't have used the term "VSCode workspace" -- just opening a folder containing multiple Cargo projects is enough. rust-analyzer can handle a workspace containing multiple separate crate graphs using different rustc versions just fine, but if the rust toolchain is being overridden in one of the subfolders your approach would not work.

Ah, I see what you mean. I've never seen r-a work in that scenario but I believe you.

Let me then withdraw that section and emphasize that, although the suggestions I'm making don't address all concerns in an ideal way, it doesn't make anything worse, and it makes the "one cargo workspace" situation much better with minimal configuration (and very few code changes / coordination requires across rust projects).

[fasterthanlime]

And that it fixes the used rust-analyzer version to the nightly version if you want nightly support.

Forgot to address that: that's not an inherent limitation of my proposal. You could have a separate setting for the "proc macro server command" — you would then use the main command of the bundled rust-analyzer (up-to-date with new features) and the proc-macro subcommand of the rustup component binary.

(This requires versioning the proc-macro-api JSON interface to be robust)

[fasterthanlime]

One remark about this: what's being merged upstream is the stable release, so if any upstream changes make it fail to build, it will be fixed at the earliest one week later (in the next stable version) + 2 days or so to get merged.

Even if nothing changes about the merge process, I'd still see it as a win. API breakage would show up early in rustc's own CI builds, giving r-a maintainers a heads up to upgrade the "sysroot" ABI before it lands in stable (what gets merged upstream).

But let me repeat that: even if no extra work is done, I still think my proposal is a win for anyone relying on rustc nightly + r-a. Because at least you know which versions do work, and they don't break retroactively.

[edwin0cheng]

Provide a different setting: "rust-analyzer.server.fromRustup": true

Is it possible only run on rust-analyzer proc-macro , but not the normal rust-analyzer ?

It is because we don't need the full rust-analyzer matches the rustc version, but only the proc abi part.

[flodiebold]

Is it possible only run on rust-analyzer proc-macro , but not the normal rust-analyzer ?

There is actually already a rust-analyzer.procMacro.server setting that would pretty much do that.