-
Notifications
You must be signed in to change notification settings - Fork 424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crate_universe
: Make module extension restarts less expensive.
#2691
Conversation
Help with CI would be appreciated, I can't spot the problem right now. |
This PR reduces the number of expensive module extension implementation restarts in combination with multiple from_cargo configurations. As each configuration processing calls module_ctx.path (which can and will cause restarts), we executed `_generate_hub_and_spokes` (i.e. `cargo-bazel) a lot and then threw that away on the next restart. Triggering `module_ctx.path` early is therefore a significant performance optimization.
a664575
to
ed10465
Compare
I significantly reworked the implementation of this, as the previous iteration would've still caused expensive restarts with from_cargo was used in different modules. The same CI job fails, and I still don't understand why that is. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems reasonable, thanks!
The CI job should be fixed by #2688 - should get that merged and this unblocked soon :)
I checked with the bazel devs, and |
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [rules_rust](https://togithub.com/bazelbuild/rules_rust) | http_archive | minor | `0.46.0` -> `0.47.1` | --- ### Release Notes <details> <summary>bazelbuild/rules_rust (rules_rust)</summary> ### [`v0.47.1`](https://togithub.com/bazelbuild/rules_rust/releases/tag/0.47.1) [Compare Source](https://togithub.com/bazelbuild/rules_rust/compare/0.46.0...0.47.1) ### 0.47.1 #### Bzlmod ```python bazel_dep(name = "rules_rust", version = "0.47.1") ``` #### WORKSPACE ```python load("@​bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") http_archive( name = "rules_rust", integrity = "sha256-35cwdTOaqqu4y+aXgIUU2C2PAKMz4+uyJ7/UMIGCmFs=", urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.47.1/rules_rust-v0.47.1.tar.gz"], ) ``` Additional documentation can be found at: https://bazelbuild.github.io/rules_rust/#setup #### What's Changed - Remote vendor doesn't require srcs present by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2688](https://togithub.com/bazelbuild/rules_rust/pull/2688) - Add flag to refer to a `sh_toolchain` for process wrapper bootstrap shebangs by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2694](https://togithub.com/bazelbuild/rules_rust/pull/2694) - Added Rust 1.79.0 by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2695](https://togithub.com/bazelbuild/rules_rust/pull/2695) - `crate_universe`: Make module extension restarts less expensive. by [@​criemen](https://togithub.com/criemen) in [https://github.com/bazelbuild/rules_rust/pull/2691](https://togithub.com/bazelbuild/rules_rust/pull/2691) - Add example of using JNI to call into Rust by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2690](https://togithub.com/bazelbuild/rules_rust/pull/2690) - Apply extra_rustc_flags list to all toolchains by [@​nmattia](https://togithub.com/nmattia) in [https://github.com/bazelbuild/rules_rust/pull/2693](https://togithub.com/bazelbuild/rules_rust/pull/2693) - rust-analyzer: Factor aliases into the project by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2709](https://togithub.com/bazelbuild/rules_rust/pull/2709) - crate_universe docs: Split bzlmod from workspace by [@​illicitonion](https://togithub.com/illicitonion) in [https://github.com/bazelbuild/rules_rust/pull/2700](https://togithub.com/bazelbuild/rules_rust/pull/2700) - Updated external dependencies docs by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2711](https://togithub.com/bazelbuild/rules_rust/pull/2711) - Update wasm-bindgen to 0.2.92 by [@​alexkirsz](https://togithub.com/alexkirsz) in [https://github.com/bazelbuild/rules_rust/pull/2714](https://togithub.com/bazelbuild/rules_rust/pull/2714) - upstream wrappers for cargo, rustc, cargo-clippy by [@​kolloch](https://togithub.com/kolloch) in [https://github.com/bazelbuild/rules_rust/pull/2703](https://togithub.com/bazelbuild/rules_rust/pull/2703) - Clean up downloads that go through the API instead of the CDN by [@​ograff](https://togithub.com/ograff) in [https://github.com/bazelbuild/rules_rust/pull/2718](https://togithub.com/bazelbuild/rules_rust/pull/2718) - Updated rust-analyzer to use repo vs generated files as crate roots by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2717](https://togithub.com/bazelbuild/rules_rust/pull/2717) - crate-universe allows workspaces with MODULE.bazel only by [@​matt-duch](https://togithub.com/matt-duch) in [https://github.com/bazelbuild/rules_rust/pull/2720](https://togithub.com/bazelbuild/rules_rust/pull/2720) - Release 0.47.0 by [@​UebelAndre](https://togithub.com/UebelAndre) in [https://github.com/bazelbuild/rules_rust/pull/2725](https://togithub.com/bazelbuild/rules_rust/pull/2725) - Update bindgen.bzl to use toolchain system include directories as well by [@​hlopko](https://togithub.com/hlopko) in [https://github.com/bazelbuild/rules_rust/pull/2727](https://togithub.com/bazelbuild/rules_rust/pull/2727) - Use newer macos runners by [@​ograff](https://togithub.com/ograff) in [https://github.com/bazelbuild/rules_rust/pull/2732](https://togithub.com/bazelbuild/rules_rust/pull/2732) #### New Contributors - [@​nmattia](https://togithub.com/nmattia) made their first contribution in [https://github.com/bazelbuild/rules_rust/pull/2693](https://togithub.com/bazelbuild/rules_rust/pull/2693) - [@​kolloch](https://togithub.com/kolloch) made their first contribution in [https://github.com/bazelbuild/rules_rust/pull/2703](https://togithub.com/bazelbuild/rules_rust/pull/2703) - [@​matt-duch](https://togithub.com/matt-duch) made their first contribution in [https://github.com/bazelbuild/rules_rust/pull/2720](https://togithub.com/bazelbuild/rules_rust/pull/2720) **Full Changelog**: bazelbuild/rules_rust@0.46.0...0.47.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/bazel-contrib/toolchains_llvm). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjUuMSIsInVwZGF0ZWRJblZlciI6IjM3LjQyNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIiwibGFiZWxzIjpbXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR reduces the number of expensive module extension implementation restarts in combination with multiple
from_cargo
configurations.As each configuration processing calls
module_ctx.path
(which can and will cause restarts), we executed _generate_hub_and_spokes (i.e.cargo-bazel
) a lot and then threw that away on the next restart.Triggering
module_ctx.path
early is therefore a significant performance optimization.On our repository, this gives us a 10sec speedup on module extension processing (local M1 mac, nothing happening in parallel), see our MODULE.bazel if you're interested in what we're doing.
This excessive restarting also exposed an upstream bazel bug on Windows 2019, where bazel spuriously fails to clean up the working directory (c.f. bazelbuild/bazel#22710).