Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Observing restarts when fetching external dependencies #1399

Closed
aignas opened this issue Sep 4, 2023 · 0 comments · Fixed by #1400
Closed

Observing restarts when fetching external dependencies #1399

aignas opened this issue Sep 4, 2023 · 0 comments · Fixed by #1400

Comments

@aignas
Copy link
Collaborator

aignas commented Sep 4, 2023

🐞 bug report

Affected Rule

whl_library

The issue is caused by the rule:

Is this a regression?

This may be an old bug as the code hasn't changed in a while.

Yes, the previous version in which this bug was not present was: ....

Description

A clear and concise description of the problem...

The whl_library restarts for a lot of times before succeeding to fetch the repo causing unnecessary load. This issue is very similar to what rules_go and gazelle have seen in

🔬 Minimal Reproduction

This can be reproduced by:

$ vim python/pip_install/pip_repository.bzl # add a print statement to better see the restarts
$ cd examples/bzlmod
$ bazel clean --expunge --async && bazel build //entry_points:yamllint
INFO: Starting clean.
INFO: Output base moved to /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025_tmp_6111_adc3dcf2-9c36-468a-87c6-10b180e635fb for deletion
Starting local Bazel server and connecting to it...
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/extensions/python.bzl:44:10: WARNING: Ignoring toolchain 'python_3_9' from module 'other_module': Toolchain 'python_3_9' from module 'example_bzlmod' already registered Python version 3.9 and has precedence
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/extensions/python.bzl:44:10: WARNING: Ignoring toolchain 'python_3_11' from module 'other_module': Toolchain 'python_3_11' from module 'rules_python' already registered Python version 3.11 and has precedence
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_yamllint
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_setuptools
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_pyyaml
DEBUG: /private/var/tmp/_bazel_aignas/ee423e5d9d08c035dfd0b914eadb5025/external/rules_python~override/python/pip_install/pip_repository.bzl:619:10: Fetching whl_library: {} rules_python~override~pip~pip_39_pathspec
INFO: Analyzed target //entry_points:yamllint (79 packages loaded, 3063 targets configured).
INFO: Found 1 target...
Target //entry_points:yamllint up-to-date:
  bazel-bin/entry_points/rules_python_entry_point_yamllint.py
  bazel-bin/entry_points/_yamllint
INFO: Elapsed time: 50.036s, Critical Path: 5.67s
INFO: 14 processes: 13 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 14 total actions
aignas@FA21050003 ~/src/github/aignas/rules_python/examples/bzlmod main

🌍 Your Environment

Running on OSX with bazelisk invoking bazel within rules_python examples/bzlmod example.

$ bazel version
Bazelisk version: development
Build label: 6.0.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Dec 19 16:07:42 2022 (1671466062)
Build timestamp: 1671466062
Build timestamp as int: 1671466062
@aignas aignas mentioned this issue Sep 4, 2023
Merged
github-merge-queue bot pushed a commit that referenced this issue Sep 4, 2023
@aignas
fix(whl_library): avoid unnecessary repository rule restarts (#1400)
6461a69 
Put the `PYTHONPATH` entries used in wheel building as a default value
to a
private attribute of the `whl_library` repository rule and use resolved
path of
the interpreter target in creating execution environment to avoid
repository
rule restarts when fetching external dependencies.

The extra private attribute on the `whl_library` removes all but one
restart
and the extra refactor removes the last restart observed when running,
which
also reduces the total execution time from around 50s to 43s on my
machine:
```console
$ cd examples/bzlmod
$ bazel clean --expunge --async && bazel build //entry_points:yamllint
```

Fixes #1399
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(whl_library): avoid unnecessary repository rule restarts aignas/rules_python
1 participant
@aignas