build probes confused by cargo always setting TARGET, even for build scripts of host dependencies

[RalfJung]

I just spent a about an hour debugging a build failure in the rustc workspace that boils down to this issue:

• the anyhow build script uses a build probe to determine if it can use std::backtracke::Backtrace.
• if the TARGET env var is set, it runs the build probe with --target passed to rustc, meaning this is conceptually a target build (this is the right thing to do, it is needed to make build probes work in cross-compilation settings when anyhow is used by the target code)
• however, since anyhow is a dependency of a build dependency, we actually need to know if the build probe would succeed on the host

In the case of bootstrap, the target sysroot has the backtrace feature stable but the host (beta) sysroot does not. So this means that anyhow enables the backtrace feature and later fails to use the backtrace feature.

I think it is a cargo bug that the TARGET env var is set for a build script that does not run on the target. Cargo should unset that env var for host build script builds.

[RalfJung]

This seems related to #5755, but there it sounds like a feature request to provide more information -- I think cargo is actively providing false information right now by setting that TARGET env var when this is in reality a build for the host.

[weihanglo]

Sorry I had a hard time reproducing this issue. Could you tell me which command you ran failed?

[RalfJung]

I ran x.py test src/tools/miri --stage 0 in rustc, in the master branch from around the time the issue was reported.

To reproduce, you'll need a proc-macro that has a dependency that has a build script.

[RalfJung]

Here's an example project. I would expect the proc macro build script to not have a TARGET env var, since it always is a host build. cargo build should therefore works. But it currently fails, showing that the TARGET env var is present.

The real problem isn't the proc macro build script itself, but build scripts in proc macro dependencies -- those can't know that they are actually being built for the host.

Though to be fair, I realized the TARGET variable is set to the host architecture even for cross-builds. In fact TARGET is set even for non-cross regular builds. However there are situations where "host and target" have the same architecture but still need to be distinguished, in particular for rustc bootstrapping (where "host" is the downloaded beta toolchain and "target" is the newly built compiler). Usually cargo (maybe by accident) supplies enough information to make that work: target crates are built with rustc --target (explicitly indicating the host architecture, so the flag is a NOP for rustc, but it is present and interpreted by the rustc wrappers that bootstrap injects), host crates do not get a --target parameter, therefore bootstrap knows which toolchain to set. But this information is lost when you add build scripts to the mix, since the TARGET env var is always set. Miri uses a similar technique to distinguish between crates that need to be built and run on the host and crates that need to be interpreted: the target string is the same in both cases, but the mere presence of --target lets Miri distinguish "host" and "target".

This is definitely kind of a hack, but AFAIK cargo provides no alternative way to do this kind of bootstraping.

[bjorn3]

However there are situations where "host and target" have the same architecture but still need to be distinguished, in particular for rustc bootstrapping

I think it would make more sense to add a new env var for this usecase.

[RalfJung]

The problem with using a new env var is that then we'll also have to adjust anyhow, autocfg, and many other crates that have logic for "build a test file with rustc to see if we can use some feature".

[weihanglo]

Back to the original issue. Did you mean that TARGET of anyhow shows a wrong specified --target info? It is expected to show host info as it will be run on host, right?

Here is a repro example. I cannot reproduce. Not sure what I missed. 😞

[jschwe]

Back to the original issue. Did you mean that TARGET of anyhow shows a wrong specified --target info? It is expected to show host info as it will be run on host, right?

@weihanglo The original issue RalfJung raised was that TARGET should not be set at all for a build script.

I think it is a cargo bug that the TARGET env var is set for a build script that does not run on the target. Cargo should unset that env var for host build script builds.

[weihanglo]

@jschwe That's a proposal they gave, not the problem they encountered.

I think it is a cargo bug that the TARGET env var is set for a build script that does not run on the target.

I interpret it as: TARGET set on anyhow should be TARGET=<HOST> but they got TARGET=<from --target>, no?

[bjorn3]

What is currently happening is that build scripts will get a TARGET set to whatever target the associated crate will be built for. What @RalfJung want is that it is unset if the associated crate is built for the host. This would be a breaking change. For example abi-cafe depends on it always being set to access the triple used for building abi-cafe to run rustc at runtime with the same triple as target.

[RalfJung]

I interpret it as: TARGET set on anyhow should be TARGET=<HOST> but they got TARGET=<from --target>, no?

No. TARGET on anyhow should be unset.

That is needed to make the existing logic work, anyway. If TARGET needs to be always set then we need to first provide some other way to convey this information from cargo to the build script (that is #5755) and then we need to change all the crates out there that run a 'probe build' to see if they can use some feature. :/ (anyhow, autocfg, thiserror, eyre -- we'll never be able to identify them all)

[RalfJung]

@bjorn3 TARGET would be always set for the 'target' part of the build (which I think is the whole build when --target is not set, since cargo then does not even properly distinguish build and runtime deps AFAIK?), so this is only a breaking change if abi-cafe is ever used as a build-dependency or from a proc macro (and someone cross-compiles that entire thing). Which I doubt is the intended usecase for that crate anyway. So I don't think that is actually a breaking change here.