Unpredictable output path for generated files

[emartinfigma]

We are trying to integrate cxx into a larger CMake-based build system.

Following some other examples we have a build.rs that contains:

fn main() {
    let _ = cxx_build::bridge("src/lib.rs");
}

Ultimately cmake generates a command line that looks like:

cargo build --target-dir /.../build/emscripten-1.40.1/rust --target=wasm32-unknown-emscripten --release

What's super confusing here is that, when executed by Ninja (which prints the command line it's running), that ends up writing lib.rs.cc/.h under

build/emscripten-1.40.1/rust/cxxbridge/...

but if I copy paste the same command from the command line, it puts them under

build/emscripten-1.40.1/rust/wasm32-unknown-emscripten/cxxbridge/...

In both cases, they contain a directory tree like

├── cxxbridge
│   ├── fullscreen-link (our crate name)
│   │   └── src
│   │       ├── lib.rs.cc -> ...symlink to a file in the build output that has a hash in the name...

For the life of me I cannot figure out what is the difference between how these two commands are run. It must be obeying some environment variable or something? Could the names of the path components leading up to the input or output dir matter? What else can I investigate to try figure out the cause of this?

[dtolnay]

I don't know of anything in this crate which would behave that way. I bet it's Cargo doing it. Notice the following:

$ cargo new --lib repro
     Created library `repro` package

$ cd repro
$ cargo build
   Compiling repro v0.1.0
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s

$ find target -name librepro.rlib
target/debug/librepro.rlib

$ cargo clean
$ cargo build --target=wasm32-unknown-emscripten
   Compiling repro v0.1.0
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s

$ find target -name librepro.rlib
target/wasm32-unknown-emscripten/debug/librepro.rlib

which is a very similar pattern.

[emartinfigma]

Yes, the --target flag reliably puts the Rust outputs in the wasm32-unknown-emscripten subdir for me. The real mystery here is how the cxx-generated outputs lib.rs.cc etc. seem to either be in that subdir or not for reasons I cannot gauge in my execution environment.

Digging more I found the discussion here maybe relevant: rust-lang/cargo#6100 , though perhaps the out dir is dir with hashes mentioned above.

I filed this bug in the hope you'd recognize it and say "oh yes that is X", but if you don't, then I agree this is likely some cargo thing out of your hands, so feel free to close.

I think to work around it on our side we could try using cxxbridge-cmd instead, does that make sense to you?

[emartinfigma]

I think https://doc.rust-lang.org/cargo/guide/build-cache.html are the relevant Rust docs about this, but they don't mention what happens with your build script generating files.

[dtolnay]

The cxxbridge CLI seems appropriate. The cxx-build crate is definitely intended only for pure Cargo managed builds, so plumbing its outputs into a CMake managed build is not an endorsed use case.

As a workaround it's possible that making your build system use CARGO_TARGET_DIR=/absolute/path/to/.../build/emscripten-1.40.1/rust as an environment variable instead of with a --target-dir flag would give more predictable behavior, as that's visible to all transitive commands dispatched by Cargo instead of just to the top level Cargo command.

[emartinfigma]

Thanks for the tip! I modified my build.rs to log CARGO_TARGET_DIR to stderr, and confirmed that it seemed to be either set or unset depending on outer configuration stuff (like which cmake generator we were using...?), and toggling between the env variable and command-line param seems to have resolved the instability.

I appreciate your concern that these paths are effectively "private API", and will put moving to using cxx-build on my mental roadmap. I briefly experimented with it but it seemed relatively costly at build time, because we need to be careful to ensure the cxx-build binary is built with the same build configuration (build output dirs / build configuration) as our other rust code, or else it builds its own copy of all dependent crates. So while it does seem more semantically correct, it also seems a little more fiddly to figure out. But that is my problem, not yours, and thanks for the help!

[Be-ing]

@emartinfigma I have a proposal for resolving this which I have described in #462. Let's continue the discussion there.