Add a --compile-time-deps build flag to cargo build

[Veykril]

Rendered

[Veykril]

cc @rust-lang/wg-rls-2 @Undin @vlad20012

[riking]

Which of the following spellings should be preferred: cargo test --compile-time-deps or cargo build --cfg=test --compile-time-deps ? If the latter, I think it's clear that this is a flag that only makes sense on build.

[Veykril]

cargo test builds and executes tests which isn't really relevant to the flag so yes if this ends up as a flag it would only really make sense on the build command

[matklad]

To build the tests today, the appropriate command is cargo test —-no-run. As we can have a proc-macro which is in dev-dependencies, combining no-run with compile-tile-deps seems like a reasonable thing to do for an IDE to expand test-specific macros.

In practice though, build —all-targets should cover this case I think.

[Aaron1011]

This will also be useful when builder Docker images - cargo build --compile-time-deps could be used to cache dependencies in a separate layer, without resorting to workaround like https://stackoverflow.com/questions/58473606/cache-rust-dependencies-with-docker-build

[bjorn3]

For Docker caching you did want to build all dependencies outside of the workspace, right? Not just build dependencies, proc macros and run build scripts for any package including those in the workspace as this RFC proposes AFAICT.

[epage]

For docker, see rust-lang/cargo#2644, particularly the summary, for why this or most solutions will not work.

[epage]

Do the messages, combined with knowing what a workspace member is, convey enough information that these can instead be filtered?

[Veykril]

Whether something is a workspace member or not is irrelevant, we'd need to be able to know whether a diagnostic comes from building a build-script / proc-macro crate. We'd also still need to rely on the unstable keep-going flag. Turns out that one is stable now, nice.

So assuming we can do that filtering, that would be an option, though this would still be slower than the current wrapper approach / a future way to only build / run the build dependencies.

[epage]

The two concepts we talk to users about build scripts and proc macros are

• build dependencies
• building for the host

[epage]

I take it this means that it would exclusively build proc macro direct dependencies and build scripts for root crates (user-selected crates, implicitly or explicitly), and all of the dependencies necessary for doing that?

Would be good to be more explicit

[RalfJung]

Here's what it causes next: build cache corruption.
rust-lang/cargo#13485

[epage]

If you are using the json output, couldn't you determine that the error is not relevant and ignore it?

[Veykril]

Kind of? We'd need to walk the reverse dependency tree for each diagnostic to check if its part of a build dependency to ignore it. I think. As we do want to report if the build script fails to compile (and hence run)

[epage]

We discussed this in today's cargo meeting

Would r-a be able to use this as a never-to-stabilize option? It feels like there is another feature hinted at by this need and it would help to accumulate use cases over time in dealing with package / built-target selection to better understand how to serve that need. We don't want to block the improvement of IDEs until that happens though.

We are sympathetic to the use case and would like to see a solution. This has the benefit of not boxing us into a corner with the build graph because of how specialized it is. However, that specialization is its own hurdle when it comes to the user experience.

• As we add features to cargo, they lower the value of all other features. Each feature makes it harder to be aware of other features, making it so people are less likely to use any of them. The more specialized a feature is, the less pay off we get for adding it and is more likely there will be additional use cases it won't satisfy, requiring us to add more
• How do we explain this to users?
  • This is three things in one: package selection (direct dep proc-macros), package selection + build target selection (build scripts of workspace members), running of build scripts of workspace members but without affecting the "create the build graph based on all the normal flags". One indication of this problem is the question "what help heading does this belong under?".
  • When a user sees this in the help, how are they suppose to know this is relevant to them?

[lnicola]

Yes, we already use a number of unstable features (i.e. RUSTC_BOOTSTRAP, __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS), so it shouldn't be a problem if it doesn't get stabilized in the medium (or even long) term.

[epage]

In that case and if @Veykril, as the RFC author, is on board, then we can proceed with this as an issue in the Cargo repo, rather than an RFC. Assumes its marked as "accepted" (I'll mark it as such as soon as someone creates it).

[vlad20012]

Important thing is that there should be a way to use this unstable feature on a stable toolchain without RUSTC_BOOTSTRAP=1. Another environment variable is an option

[RalfJung]

That's not how unstable features work?

[lnicola]

I guess one concern is how it might interact with feature detection done by build scripts.

An unstable Cargo feature might be better anyway.

[bjorn3]

I guess one concern is how it might interact with feature detection done by build scripts.

For that rust-analyzer could use RUSTFLAGS="-Zallow-feature=" I think when RUSTC_BOOTSTRAP isn't set already. Not every build script handles this fine, but those that don't are broken anyway.

[Veykril]

I guess one concern is how it might interact with feature detection done by build scripts.

For that rust-analyzer could use RUSTFLAGS="-Zallow-feature=" I think when RUSTC_BOOTSTRAP isn't set already. Not every build script handles this fine, but those that don't are broken anyway.

Won't that cause rebuilds? Also that itself is a Z flag won't that still need a nightly channel?

In that case and if @Veykril, as the RFC author, is on board, then we can proceed with this as an issue in the Cargo repo, rather than an RFC. Assumes its marked as "accepted" (I'll mark it as such as soon as someone creates it).

That is fine by me, that will at least make things better on nightly toolchains. I wasn't expecting this RFC to become stable that quickly either way (there is a lot of use cases to explore still as you mentioned). Using it on non-nightly toolchains is a good question though since we can't use RUSTC_BOOTSTRAP=1 thanks to build script probes :/

I guess we could repurpose the RUSTC_WRAPPER and instead of its cursed functionality merely have it strip the RUSTC_BOOTSTRAP flag from invocations.

[bjorn3]

Won't that cause rebuilds?

Right, it does.

Also that itself is a Z flag won't that still need a nightly channel?

If you set RUSTC_BOOTSTRAP=1 it works on stable and causes #![feature] to be disabled entirely even when using RUSTC_BOOTSTRAP or nightly.

[lnicola]

Shouldn't this be a Cargo feature anyway? Those don't need RUSTC_BOOTSTRAP=1.

[epage]

Cargo checks RUSTC_BOOTSTRAP

[Veykril]

Opened rust-lang/cargo#14434

[vlad20012]

Some build scripts emit cargo::rerun-if-changed=RUSTC_BOOTSTRAP. Namely, this does anyhow, that is a dependency of proc_macro2, that is a dependency of basically any non-trivial rust project.

If rust-analyzer invokes cargo with RUST_BOOTSTRAP=1 in a project with such a build script, the project will be rebuilt almost from scratch during the next cargo build invocation without RUSTC_BOOTSTRAP=1.

Basically, a project would be rebuilt from scratch after opening in rust-analyzer.

This is why requiring RUSTC_BOOTSTRAP=1 for this feature is very unsatisfactory.

May we have another hacky variable specially for this use case?

[Veykril]

May we have another hacky variable specially for this use case?

Looks like we might already: rust-lang/cargo#14434 (comment)

I will close this RFC then in favor of that cargo issue as the place for discussion