No repository visible as '@python_3_11_x86_64-apple-darwin' from repository '@@_main~dev~rules_testing_dev_toolchains' error with Bazel@HEAD

[sgowroji]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3461#018bd123-ec1e-43cb-aad6-7c8b6c6c2203

Platform : Bzlmod on :darwin: (OpenJDK 11, Xcode)), (Bzlmod on :windows: (OpenJDK 11, VS2022))

Logs:

ERROR: no such package '@@[unknown repo 'python_3_11_x86_64-apple-darwin' requested from @@_main~dev~rules_testing_dev_toolchains]//': The repository '@@[unknown repo 'python_3_11_x86_64-apple-darwin' requested from @@_main~dev~rules_testing_dev_toolchains]' could not be resolved: No repository visible as '@python_3_11_x86_64-apple-darwin' from repository '@@_main~dev~rules_testing_dev_toolchains'
(05:04:00) ERROR: /private/var/tmp/_bazel_buildkite/9fc99e1aa7f3f63f7564fddf677bd6a4/external/bazel_tools/tools/jdk/BUILD:133:10: Target '@@bazel_tools//tools/jdk:proguard_whitelister' depends on toolchain '@@[unknown repo 'python_3_11_x86_64-apple-darwin' requested from @@_main~dev~rules_testing_dev_toolchains]//:python_runtimes', which cannot be found: no such package '@@[unknown repo 'python_3_11_x86_64-apple-darwin' requested from @@_main~dev~rules_testing_dev_toolchains]//': The repository '@@[unknown repo 'python_3_11_x86_64-apple-darwin' requested from @@_main~dev~rules_testing_dev_toolchains]' could not be resolved: No repository visible as '@python_3_11_x86_64-apple-darwin' from repository '@@_main~dev~rules_testing_dev_toolchains''

Steps:

git clone https://github.com/bazelbuild/rules_testing.git
git reset b88ee3c9d456f9d812ab6748aaaa709e8eb369c4 --hard
export USE_BAZEL_VERSION=51fd503c634d868dadbdc5b532da9d979119afb2
bazel test  --enable_bzlmod --test_tag_filters=-skip-bzlmod,-docs  --lockfile_mode=off --build_tag_filters=-no_bazel_downstream --test_tag_filters=-no_bazel_downstream -- ...

Note: Not able to figure out the bisect culprit.

CC Greenteam @SalmaSamy @Wyverald

[sgowroji]

Hi @rickeylev, Can you take a look at the above issue. Thanks!

[rickeylev]

Sorry, missed when this was originally filed. I'll have a look now.

[rickeylev]

I can't reproduce on Linux. I had a co-worker repro this on his Mac.

It looks like rules_testing's CI doesn't have Mac re-enabled yet, so I can use that to try some things.

It'll take me a day or 7 to get my work mac up and running again (depends on if it still boots up or not)

[sgowroji]

Yes its failing on Ubuntu and MacOS environment.

[rickeylev]

Yes its failing on Ubuntu

Typo? I don't see the Ubuntu failures? I see windows and mac failures.

I got my mac running. I think i found the cause. I'll work on a fix.

[rickeylev]

Aha! Actually, this is technically an issue with the CI pipeline. I think the issue is --test_tag_filters=-no_bazel_downstream -- it's clobbering an earlier instance of that flag that skips targets/tests that shouldn't be run. The --test_tag_filters arg isn't additive; the last usage clobbers previous usages (this has bitten me before, its annoying and surprising)

So I'd suggest fixing that in your CI pipeline, as it's probably causing various false positives.

I only figured this out after cleaning out some hacks which do allow mac+bzlmod builds to pass, so I'll send that PR anyways.

[rickeylev]

So I'd suggest fixing that in your CI pipeline

Just to clarify: my fix will probably make the downstream CI pass, but the extra tests it's running aren't supposed to run on Mac or Windows. The two categories of tests being included are (1) ones specifically not intended for bzlmod, and (2) tests for generated documentation, which is only configured to build for, and run on, Linux. I'm surprised they pass today.

@sgowroji