Deeper rust-analyzer Integration with Non-Cargo Build Systems

[davidbarsky]

This issue connects several disparate issues and discussions, such as #13437, #13128, #12105, #13226, and several others that I'm probably missing. This issue will be a bit handy-wavy, but I think there's value in describing my ideal user experience, which is to make rust-analyzer work with Buck/Bazel/et. al. as just as well as if they were Cargo.

The Current State

My current employer has one of the largest monorepos in the world and uses the Buck build systemto manage the vast majority of Rust builds. At the moment, most users connect to a remote dev server via VS Code and have their VS Code workspace be at the root of the monorepo (these are largely immutable user behavior and expectations). They then run a small CLI that spits out a rust-project.json (e.g., rust-project develop //foo/bar/my-target) and work on their project from the root of the workspace. To get cargo check-style diagnostics, people need to manually invoke Buck with a check-flavored build as such: buck build :my-target[check].

Possible Improvements

In no particular order, since I see these as roughly equivalent in impact:

• After VS Code has activated the rust-analyzer extension (for instance, after the user has opened a Rust file), rust-analyzer-the-language-server should be able to delegate to an external tool (in my case, rust-project) that can query the build system for the owning TARGETs, generate a rust-project.json, and continue with the remainder of the rust-project.json-based workspace loading. This option would occur after rust-analyzer checks for a Cargo.toml or a rust-project.json, doesn't find them, and a "discover workspace" (name to be bikeshed!) command is defined.
• The flycheck crate should be configurable to run builds/checks through a non-Cargo-based build system. When I last looked at the flycheck crate, it didn't seem like a particularly difficult change, just one that nobody has gotten around to designing or implementing.
• All of the above features can be configured through a rust-analyzer.toml (which I don't believe has an issue), which can be placed at the root of the monorepo, which would reduce how many configuration forks I need to make for the rust-analyzer extension.

[bjorn3]

The flycheck crate should be configurable to run builds/checks through a non-Cargo-based build system. When I last looked at the flycheck crate, it didn't seem like a particularly difficult change, just one that nobody has gotten around to designing or implementing.

Does buck allow passing --error-format=json to rustc and forwarding the resulting json messages? Flycheck needs this as it can only parse error messages in rustc's json format. The default user facing error message format of rustc is unstable, non-trivial to parse and I believe misses some information that rust-analyzer needs.

[davidbarsky]

Does buck allow passing --error-format=json to rustc and forwarding the resulting json messages? Flycheck needs this as it can only parse error messages in rustc's json format. The default user facing error message format of rustc is unstable, non-trivial to parse and I believe misses some information that rust-analyzer needs.

It does. At the moment, IDE-specific diagnostics are written to a deterministic folder in the output directory, and while I think that I'd need to convince the Buck team to revisit their stance on writing to, e.g., stderr, I feel that it's something that's pretty tractable and I could succeed on.

[Veykril]

The flycheck crate should be configurable to run builds/checks through a non-Cargo-based build system. When I last looked at the flycheck crate, it didn't seem like a particularly difficult change, just one that nobody has gotten around to designing or implementing.

What exactly do you mean here? You can already fully overwrite the command that is being executed by flycheck (the flycheck code just mentions cargo everywhere still even though its really just some command with a specific json output at this point)

Ye I believe we don't have an issue for that yet, but we probably want to have such a config at some point

[davidbarsky]

What exactly do you mean here? You can already fully overwrite the command that is being executed by flycheck (the flycheck code just mentions cargo everywhere still even though its really just some command with a specific json output at this point)

I stand corrected! I just assumed... that it didn't work. I'm seeing rust-analyzer invoke buck correctly when I point rust-analyzer to the right target and have a shells script invoke buck and read out the JSON diagnostics from rustc. The main issue is that it requires a bit of manual configuration. Would y'all accept a PR that adds a field to rust-project.json that overrides rust-analyzer.checkOnSave.overrideCommand (as the tool generating the rust-project.json has the best knowledge of what command to run to get diagnostics)?

Ye I believe we don't have an issue for that yet, but we probably want to have such a config at some point

I'm guessing that this refers to the currently hypothetical rust-analyzer.toml?

[Veykril]

Would y'all accept a PR that adds a field to rust-project.json that overrides rust-analyzer.checkOnSave.overrideCommand (as the tool generating the rust-project.json has the best knowledge of what command to run to get diagnostics)?

I feel like it would make more sense to support that setting in a rust-analyzer.toml?

I'm guessing that this refers to the currently hypothetical rust-analyzer.toml?

Yes sorry, I failed to copy the quoted text :)

[googleson78]

To chime in, because I'm working on the exact same thing (as you already linked)

cargo check

Being able to checkOnSave.overrideCommand is great and somewhat works, but we need to have finer granularity on running it, because right now it's a static command, and we can't specify "please only build this one single file and its dependencies". In bigger repos, rebuilding everything is not viable, and having to wait 30 minutes to build because a PR unrelated to you changed something entirely unrelated to you sucks.

"Run tests"

The code lenses that pop up saying Run tests currently don't get disabled even if you use rust-project.json, even though they will still try to use cargo. They should get disabled. However ideally it would be really good if we could set what command to run instead of cargo for the tests! If the "current file" variable interpolation is also implemented, I imagine that that would be good enough to provdie specific enough information to the external tool about what tests to run.

I'll write a separate issue for this.

Config

Having a specific file for this config would be great. To be honest I don't like having the configuration passed in from the LSP client, because then every dev has to manually do that. I'm not sure if this is good design however.

[davidbarsky]

I feel like it would make more sense to support that setting in a rust-analyzer.toml?

@Veykril My feeling is that it makes sense to support this feature in both places—I don't think I can share exact numbers, but at least for my employer, it's not feasible to have a single rust-analyzer.toml that defines a single build command for the entire repository. Unlike something like cargo check or cargo build, which is able to be used across projects and is able to infer what package is being built by looking at the present working directory, Buck requires specifying the exact package ("target", in Buck's parlance) to be built. It's as if Cargo required every build to be specified with cargo build -p some-package. For cultural and code organization reasons, it also doesn't really make sense to place a rust-analyzer.toml alongside each target that wants IDE support—Buck supports target trees that are arbitrarily nested (unlike Cargo workspaces, which are typically only a single level deep), and people expect to navigate across those targets and continue to receive IDE support.

For Buck users, rust-project.json would ideally become a hidden implementation detail that is entirely controlled/mediated by either Buck or tooling that integrates with Buck. Since, at the moment, the user knows what Buck target they're intending to work on (and are running the rust-project CLI accordingly), I think it makes the make sense to colocate the build command alongside the project graph. I hope that provided sufficient detail, and if not, I'm happy to elaborate any point.

---

To put what I said in a different, higher-level way: roughly speaking, rust-analyzer assumes a model where one VS Code workspace corresponds to a single editable/buildable project (this is how I work in the monorepo, but it's becoming clear that I'm the exception). A lot of folks who expect monorepo-oriented tooling have a mental model of one VS Code workspace corresponding to many, many, editable/buildable projects, only one or two of which the user interested in working on. For better or worse, rust-analyzer needs a bit of help knowing which node in this very, very, wide tree it needs to analyze and build-on-save, and a rust-project.json seems like the natural place to put that information.

(I assume that everything I said about Buck also applies to Bazel, but I've never used Bazel.)

[googleson78]

@davidbarsky Yup, it's pretty much the same. We discussed this a bit and a potential solution in #13437

[Veykril]

Yes you are right, I was mainly thinking of checkOnSave = diagnostics related config, but really it integrates like all the other build related configs, so having that be part of the rust-project.json makes sense still, at least for the override command.

[Veykril]

@davidbarsky, Okay I got one more question regarding the rust-project.json stuff in regards to overrideCommand here. Would it not be feasible for you to generate a rust-analyzer.toml along side your rust-project.json generation?

[davidbarsky]

@Veykril While it's not unreasonable to generate a rust-project.json alongside a rust-analyzer.toml Buck's equivalent of cargo metadata (buck cquery) returns absolute paths and, therefore, can't be committed since it includes people's usernames. The generated rust-project.json is also often 30-40k lines of JSON and the source control team prefers that the file wasn't committed. The build command for Buck that includes the target being built is also sensitive to the location from which the Buck is being invoked.

I think it'd be neat if the Buck integration tooling can do all of this automatically, in the background, by placing the generated rust-project.json in an Eden-managed tmpdir and communicated to rust-analyzer where to look, but that's a little bit further away.

To step back a little, I view the rust-project.json as ephemeral and per-user, whereas I see the rust-analyzer.toml as global to all rust-analyzer users in the monorepo. I'm most interested in a rust-analyzer.toml because it'd allow me to reduce how many configuration forks I need to make in vending rust-analyzer. At the moment, it's not a lot, but I'd like to be able to get it down to zero. That being said, I don't see why there can't be Cargo-style hierarchal configs with rust-project.json—I know some teams would want to set their own settings.

---

When trying to add support for loading the build command to rust-analyzer yesterday, I found I could make the change to load the rust-project.json in the main loop when rust-analyzer is checking for linked paths. It's not the cleanest change, but it'd avoid the need to change Arc<Config> to Arc<Mutex<Config>> or something similar.

[Veykril]

While it's not unreasonable to generate a rust-project.json alongside a rust-analyzer.toml Buck's equivalent of cargo metadata (buck cquery) returns absolute paths and, therefore, can't be committed since it includes people's usernames.

Would you also need to specify absolute paths for the overrideCommands? And I was thinking of generating it without committing it (whats the point of having it generated by the user otherwise).

That being said, I don't see why there can't be Cargo-style hierarchal configs with rust-project.json—I know some teams would want to set their own settings.

I don't think I'd want any config to be specifiable in the rust-project.json, that's not the purpose of that file. It's purpose is to specify the structure of a project without using cargo. That's where rust-analyzer.toml should come into play imo.

[bjorn3]

overrideCommand makes sense to me in rust-project.json as it defines how to build the project and if you have multiple projects each one is expected to have a different overrideCommand. Most other configs make sense to be specified at a user or vscode workspace level, but overrideCommand only really makes sense at a per-project level. Even when using cargo you may want to enable a cargo feature for one project (as it needs it for your purposes), but not for another project (as it doesn't have this feature at all).

[Veykril]

Yes, in general I agree that it makes sense. I mainly wanted to confirm whether rust-analyzer.toml could make sense here as well or not, as I also associate that with project setups.

I also just realized

I don't think I'd want any config to be specifiable in the rust-project.json,

can be read differently than intended, with any I did not mean I do not want any config there at all, but not just any random config. Build specific ones do make sense (which means checkOnSave.overrideCommand and buildScripts, but anything else doesn't really seem to make too much sense (I also just realized that buildScripts having the cargo prefix seems like a bad choice, we probably want to alias that one).

[davidbarsky]

Would you also need to specify absolute paths for the overrideCommands? And I was thinking of generating it without committing it (whats the point of having it generated by the user otherwise).

I'm not sure, but I'm leaning towards a "no" from necessity standpoint but "yes" for ease of supporting and debugging people's environments. And yes, agreed on the generation of the rust-project.json.

As for the second point about hierarchal rust-project.jsons—that was a typo, I meant to refer to rust-analyzer.toml instead, sorry about that!