Set CARGO_TARGET_DIR for build scripts

[dtolnay]

In public GitHub repos one can find almost infinite creative and awful ways that crates try to guess the Cargo target dir.

• The old way, which "didn't work for workspaces", and the new way, which "doesn't work for doc-tests"
• The $manifest_dir/target way, which does not work when built with --target-dir or when building as a dependency of a different crate
  • https://github.com/rust-lang/rust-clippy/blob/5af88e3c2d8cc4fb74a0e455381669930ee3a31a/tests/integration.rs#L30-L32
  • https://github.com/rust-lang/rust/blob/f3c923a13a458c35ee26b3513533fce8a15c9c05/src/tools/clippy/src/main.rs#L135-L143
  • https://github.com/cmyr/cargo-instruments/blob/27f62c0c6e6359bd8293e2697f4aabfd60d5b93e/src/instruments.rs#L119-L128
  • https://github.com/swc-project/swc/blob/24c597f097235c9756a5d8d6e1e2bdf76b3261cb/testing/src/paths.rs#L4-L19
  • https://github.com/cloudflare/quiche/blob/72a2fe96d2610d8da889713f1ce71ba3d4c1150e/src/build.rs#L170-L172
• The cargo metadata | jq .target_directory way, which does not work when built with --target-dir
  • https://github.com/gnzlbg/cargo-asm/blob/577f890ebd4a09c8265710261e976fe7bfce8668/src/target.rs#L157-L169
• The $out_dir/../../.. way, which is an assumption about how deeply nested Cargo's placement of out dirs will be in the future
  • https://github.com/budziq/rust-skeptic/blob/e271ea6acdc61fc2102af98e0415b09b539ca9af/src/skeptic/lib.rs#L783-L795
• The $current_dir/target way, which does not work when built with --target-dir or as a dependency
  • https://github.com/zombodb/pgx/blob/454eddf22d9d6e746b52698f829381170239f73f/pgx-utils/src/lib.rs#L154-L163
• The $manifest_dir/../target way, as above but hardcoding this crate's placement in its workspace
  • https://github.com/anp/moxie/blob/07638e7b9785a94216f9fdeb38ab585ca6806173/ofl/src/main.rs#L19-L22 + https://github.com/anp/moxie/blob/07638e7b9785a94216f9fdeb38ab585ca6806173/ofl/src/coverage.rs#L104
• The way that involves trying to parse .cargo/config and .cargo/config.toml before falling back to one of the previous
  • https://github.com/tauri-apps/tauri/blob/e760331fa1c93dde1465308cf42abe17a80a231e/cli/tauri-bundler/src/bundle/settings.rs#L464-L492
• The $current_exe/../.. way
  • https://github.com/conradkleinespel/rpassword/blob/c786de157b15ea326c5e21f6270b0b5ca1e498f9/tests/piped.rs#L9-L16
  • https://github.com/denoland/deno/blob/d245ececb6c3eed9d6a01b877cfbbf7642454fc3/test_util/src/lib.rs#L75-L80
• The $current_exe/../../conditional .. way

  • cargo/crates/cargo-test-support/src/paths.rs

    Lines 18 to 29 in e4b65bd

    	let mut path = t!(env::current_exe());
    	path.pop(); // chop off exe name
    	path.pop(); // chop off 'debug'
    	// If `cargo test` is run manually then our path looks like
    	// `target/debug/foo`, in which case our `path` is already pointing at
    	// `target`. If, however, `cargo test --target $target` is used then the
    	// output is `target/$target/debug/foo`, so our path is pointing at
    	// `target/$target`. Here we conditionally pop the `$target` name.
    	if path.file_name().and_then(|s| s.to_str()) != Some("target") {
    	path.pop();
    	}

Now, it's likely many of these should not be trying to poke around in the target dir in the first place, but the fact remains that there are motivating reasons for wanting to. For example, to perform some work once that will be shared/available to multiple crates' build scripts to avoid repeating the same work in each -- https://github.com/dtolnay/scratch supports this use case but it involves crates writing into a different crate's OUT_DIR, which is even more of a hack than just using a well-namespaced subdirectory of the target dir.

This PR adds an environment variable CARGO_TARGET_DIR set by Cargo for build scripts to inform them of Cargo's placement of the target dir. This provides the correct value even when your build was invoked with --target-dir or when the build script is for a crate that is not the build root so $manifest_dir/target would not be a good guess.

[rust-highfive]

r? @ehuss

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

Thanks for the PR here, and also thanks for gathering all those links together! I didn't realize there were so many crates doing so many different things to find the target directory.

I don't really know what to do about this though. As you've documented here it's not stable to rely on anything about the target directory. Given that, how can crates productively use this environment variable? This also doesn't even begin to touch on the concurrency aspects where there's no protection against crates concurrently using the target directory in different ways.

On one hand I don't want to be forced into a solution which only makes sense for a small subset of users, but I also don't want to prevent something from happening if there's a clear need for it.

I realize this may be difficult to do, but I think it would be useful to try to see if there's something else Cargo can do in a stable way to address the majority of these use cases. For example with your scratch crate if the main purpose is to work with cargo clean, maybe we could add a feature to build scripts to register a directory with Cargo that gets cleaned with cargo clean (outside the target directory). Or maybe we could reserve target/debug/scratch for all crates and set that for build scripts if all they need is shared global scratch space for the whole build.

I'd basically like to be able to comfortably document this as "please use this, it will remain stable" rather than having documentation of "here's a thing we were forced to give you and may break in the future, so while we say don't use it you're probably going to use it anyway"

[dtolnay]

Thanks Alex -- that makes perfect sense. I think I'll close this for now, look through each of those crates more carefully to categorize exactly what they need or what already supported thing they should be doing instead, and come back with an appropriate proposal.

[Eh2406]

More power to you. That is the kind of thankless mind bending work that moves the community forward. Thank you.

[weihanglo]

See also a new feature request #9661