Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid double building cargo_build_script.data targets #2826

Merged
merged 2 commits into from
Sep 10, 2024

Conversation

UebelAndre
Copy link
Collaborator

Before this change targets passed to cargo_build_script.data would get built multiple times (once per configuration)

% bazel cquery 'filter(//test/cargo_build_script/location_expansion:target_data.txt, deps(//test/cargo_build_script/location_expansion:build_rs))'
//test/cargo_build_script/location_expansion:target_data.txt (be33641)
//test/cargo_build_script/location_expansion:target_data.txt (a1c326f)

We can see that this is the exec (a1c326f) and target (be33641) configurations.

% bazel config a1c326f be33641
INFO: Displaying diff between configs a1c326f and be33641
Displaying diff between configs a1c326f and be33641
FragmentOptions com.google.devtools.build.lib.analysis.PlatformOptions {
  platforms: [@@internal_platforms_do_not_use//host:host], [@@bazel_tools//tools:host_platform]
}
FragmentOptions com.google.devtools.build.lib.analysis.config.CoreOptions {
  compilation_mode: opt, fastbuild
  is exec configuration: true, false
  platform_suffix: exec, null
}
FragmentOptions com.google.devtools.build.lib.rules.android.AndroidConfiguration$Options {
  fat_apk_cpu: [], [armeabi-v7a]
}
FragmentOptions com.google.devtools.build.lib.rules.cpp.CppOptions {
  copt: [-g0], []
  cxxopt: [-g0], []
  strip: always, sometimes
}
FragmentOptions com.google.devtools.build.lib.rules.java.JavaOptions {
  java_runtime_version: remotejdk_11, local_jdk
  jvmopt: [-XX:ErrorFile=/dev/stderr], []
}

This is caused by data being passed to the underlying rust binary in cargo_build_script_wrapper.bzl.

The fix for this ends up requiring some gymnastics as many existing uses of cargo_build_script rely on data files being available within the runfiles of cargo_build_script.script. This change accounts for that by creating a wrapper rule for the build script that symlinks the cfg=exec Rust binary (which is the actual Cargo build script) and updating the cargo_build_script rule to consume it as cfg=target. Within the cargo_build_script rule this means the script target and it's data attribute files will be in cfg=target but the binary itself will have been built for cfg=exec and will be runnable on the exec platform.

We see after this change that data targets only appear once now

% bazel cquery 'filter(//test/cargo_build_script/location_expansion:target_data.txt, deps(//test/cargo_build_script/location_expansion:build_rs))'
//test/cargo_build_script/location_expansion:target_data.txt (be33641)

@UebelAndre
Copy link
Collaborator Author

UebelAndre commented Sep 1, 2024

@scentini This change fails the compatible_with tests and I don't understand why (or really how compatible_with works), can you help me understand what's happening? 😅

% bazel test //test/cargo_build_script/... --keep_going
ERROR: /Users/rules_rust/test/cargo_build_script/compatible_with/BUILD.bazel:27:19: in cargo_build_script rule //test/cargo_build_script/compatible_with:empty_build_script: dependency //test/cargo_build_script/compatible_with:empty_build_script- doesn't support expected environment: //test/cargo_build_script/compatible_with:bar
ERROR: /Users/rules_rust/test/cargo_build_script/compatible_with/BUILD.bazel:27:19: Analysis of target '//test/cargo_build_script/compatible_with:empty_build_script' failed
WARNING: errors encountered while analyzing target '//test/cargo_build_script/compatible_with:empty_build_script', it will not be built.
INFO: Analyzed 18 targets (6 packages loaded, 47 targets configured).
WARNING: errors encountered while analyzing target '//test/cargo_build_script/compatible_with:test_compatible_with', it will not be built.
Analysis failed

@UebelAndre
Copy link
Collaborator Author

@scentini This change fails the compatible_with tests and I don't understand why (or really how compatible_with works), can you help me understand what's happening? 😅

% bazel test //test/cargo_build_script/... --keep_going
ERROR: /Users/rules_rust/test/cargo_build_script/compatible_with/BUILD.bazel:27:19: in cargo_build_script rule //test/cargo_build_script/compatible_with:empty_build_script: dependency //test/cargo_build_script/compatible_with:empty_build_script- doesn't support expected environment: //test/cargo_build_script/compatible_with:bar
ERROR: /Users/rules_rust/test/cargo_build_script/compatible_with/BUILD.bazel:27:19: Analysis of target '//test/cargo_build_script/compatible_with:empty_build_script' failed
WARNING: errors encountered while analyzing target '//test/cargo_build_script/compatible_with:empty_build_script', it will not be built.
INFO: Analyzed 18 targets (6 packages loaded, 47 targets configured).
WARNING: errors encountered while analyzing target '//test/cargo_build_script/compatible_with:test_compatible_with', it will not be built.
Analysis failed

I believe I've fixed it.

@UebelAndre UebelAndre force-pushed the double-build branch 4 times, most recently from d181682 to 0a0f254 Compare September 6, 2024 17:59
Copy link
Collaborator

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really cute, thanks for putting it together!

Are you imagining putting together any affordances for using this from crate_universe, or leaving folks to do that on their own?

@UebelAndre UebelAndre added this pull request to the merge queue Sep 10, 2024
github-merge-queue bot pushed a commit that referenced this pull request Sep 10, 2024
Before this change targets passed to `cargo_build_script.data` would get
built multiple times (once per configuration)
```
% bazel cquery 'filter(//test/cargo_build_script/location_expansion:target_data.txt, deps(//test/cargo_build_script/location_expansion:build_rs))'
//test/cargo_build_script/location_expansion:target_data.txt (be33641)
//test/cargo_build_script/location_expansion:target_data.txt (a1c326f)
```
We can see that this is the exec (`a1c326f`) and target (`be33641`)
configurations.
```
% bazel config a1c326f be33641
INFO: Displaying diff between configs a1c326f and be33641
Displaying diff between configs a1c326f and be33641
FragmentOptions com.google.devtools.build.lib.analysis.PlatformOptions {
  platforms: [@@internal_platforms_do_not_use//host:host], [@@bazel_tools//tools:host_platform]
}
FragmentOptions com.google.devtools.build.lib.analysis.config.CoreOptions {
  compilation_mode: opt, fastbuild
  is exec configuration: true, false
  platform_suffix: exec, null
}
FragmentOptions com.google.devtools.build.lib.rules.android.AndroidConfiguration$Options {
  fat_apk_cpu: [], [armeabi-v7a]
}
FragmentOptions com.google.devtools.build.lib.rules.cpp.CppOptions {
  copt: [-g0], []
  cxxopt: [-g0], []
  strip: always, sometimes
}
FragmentOptions com.google.devtools.build.lib.rules.java.JavaOptions {
  java_runtime_version: remotejdk_11, local_jdk
  jvmopt: [-XX:ErrorFile=/dev/stderr], []
}
```

This is caused by `data` being passed to the underlying rust binary in
`cargo_build_script_wrapper.bzl`.

The fix for this ends up requiring some gymnastics as many existing uses
of `cargo_build_script` rely on `data` files being available within the
runfiles of `cargo_build_script.script`. This change accounts for that
by creating a wrapper rule for the build script that symlinks the
`cfg=exec` Rust binary (which is the actual [Cargo build
script](https://doc.rust-lang.org/cargo/reference/build-scripts.html))
and updating the `cargo_build_script` rule to consume it as
`cfg=target`. Within the `cargo_build_script` rule this means the script
target and it's `data` attribute files will be in `cfg=target` but the
binary itself will have been built for `cfg=exec` and will be runnable
on the exec platform.

We see after this change that data targets only appear once now
```
% bazel cquery 'filter(//test/cargo_build_script/location_expansion:target_data.txt, deps(//test/cargo_build_script/location_expansion:build_rs))'
//test/cargo_build_script/location_expansion:target_data.txt (be33641)
```
@UebelAndre
Copy link
Collaborator Author

This is really cute, thanks for putting it together!

Are you imagining putting together any affordances for using this from crate_universe, or leaving folks to do that on their own?

What do you mean? This change shouldn't require any interface changes so any direct uses of cargo_build_script (wrapper/top level symbol) should start using this.

Merged via the queue into bazelbuild:main with commit cbdf540 Sep 10, 2024
4 checks passed
@illicitonion
Copy link
Collaborator

This is really cute, thanks for putting it together!
Are you imagining putting together any affordances for using this from crate_universe, or leaving folks to do that on their own?

What do you mean? This change shouldn't require any interface changes so any direct uses of cargo_build_script (wrapper/top level symbol) should start using this.

I mean right now there are annotations like build_script_data which generate attributes on a cargo_build_script target which is auto-generated for the crates - do we want to change what gets generated in the BUILD files?

github-merge-queue bot pushed a commit that referenced this pull request Sep 11, 2024
…ata (#2855)

#2826 ended up introducing
a regression for crates with build scripts that require extra data at
compile time. This change adds a default glob for all
`cargo_build_script` targets generated by `crate_universe` and also
introduces an `annotation.build_script_compile_data` attribute to add
custom targets to generated outputs.
github-merge-queue bot pushed a commit that referenced this pull request Oct 5, 2024
After #2826 was merged, I
started seeing flaky builds on Windows related to build script
executables
(#2891 (comment)).
This appears to be related to
bazelbuild/bazel#21747 so to avoid the issue,
this change ensures that Windows build script executables are copied
instead of symlinked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants