pipstar regression: Labels duplicated in whl_library_targets_from_requires

[thirtyseven]

🐞 bug report

Affected Rule

The issue is caused by the rule: pip.parse

Is this a regression?

Yes, the previous version in which this bug was not present was: 1.7.0

Description

Some pip packages fail to build in 1.8.0-rc3 with errors like:

ERROR: /private/var/tmp/_bazel_tkaplan/2ee08b28e31e216c71c3665814b9e6fa/external/rules_python++pip+py3deps_310_ray_cp310_cp310_macosx_11_0_arm64_6f221b14/BUILD.bazel:6:34: in deps attribute of py_library rule @@rules_python++pip+py3deps_310_ray_cp310_cp310_macosx_11_0_arm64_6f221b14//:pkg: Label '@@rules_python++pip+py3deps//grpcio:pkg' is duplicated. Since this rule was created by the macro 'whl_library_targets_from_requires', the error might have been caused by the macro implementation
ERROR: /private/var/tmp/_bazel_tkaplan/2ee08b28e31e216c71c3665814b9e6fa/external/rules_python++pip+py3deps_310_ray_cp310_cp310_macosx_11_0_arm64_6f221b14/BUILD.bazel:6:34: Analysis of target '@@rules_python++pip+py3deps_310_ray_cp310_cp310_macosx_11_0_arm64_6f221b14//:pkg' (config: 854dc54) failed

🔬 Minimal Reproduction

MODULE.bazel:

bazel_dep(name = "rules_python", version = "1.8.0-rc3")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.10",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    enable_implicit_namespace_pkgs = True,
    hub_name = "pypi",
    python_version = "3.10",
    requirements_lock = "requirements_mac.txt",
)
use_repo(pip, "pypi")

uv = use_extension(
    "@rules_python//python/uv:uv.bzl",
    "uv",
    dev_dependency = True,
)
uv.configure(version = "0.7.12")

BUILD.bazel:

load("@rules_python//python/uv:lock.bzl", "lock")

DEFAULT_UV_ARGS = [
    "--index-strategy=unsafe-best-match",
]

# Generate lock files for default Python version
lock(
    name = "requirements_mac",
    srcs = [":requirements.in"],
    out = "requirements_mac.txt",
    args = DEFAULT_UV_ARGS + ["--python-platform=macos"],
)

lock(
    name = "requirements_linux",
    srcs = [":requirements.in"],
    out = "requirements_linux.txt",
    args = DEFAULT_UV_ARGS + ["--python-platform=linux"],
)

requirements.in:

ray[client]==2.44.1

Run:

bazel run :requirements.update
bazel build @pypi//...

🔥 Exception or Error

bazel build @pypi//...     
ERROR: /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_ray/BUILD.bazel:6:34: in data attribute of filegroup rule @@rules_python++pip+pypi_310_ray//:whl: Label '@@rules_python++pip+pypi//grpcio:whl' is duplicated. Since this rule was created by the macro 'whl_library_targets_from_requires', the error might have been caused by the macro implementation
ERROR: /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_ray/BUILD.bazel:6:34: Analysis of target '@@rules_python++pip+pypi_310_ray//:whl' (config: 7f8856b) failed
Use --verbose_failures to see the command lines of failed build steps.
ERROR: Analysis of target '@@rules_python++pip+pypi//ray:whl' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.578s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

🌍 Your Environment

Operating System:

  
Mac OS 15.7.3
  

Output of bazel version:

  
8.5.1
  

Rules_python version:

  
1.8.0-rc3
  

Anything else relevant?
The error does not occur with --repo_env=RULES_PYTHON_ENABLE_PIPSTAR=0

[rickeylev]

I followed the repro steps, but it didn't reproduce.

I had to add

uv.configure(
    version = "0.6.2",
)

to the module.bazel to make it work.

Can you share the requirements_lock.txt? Perhaps this is mac-specific in some way?

[rickeylev]

In any case, if this bug is legit, then a simple fix is probably to change this line:

https://github.com/bazel-contrib/rules_python/blob/main/python/private/pypi/whl_library_targets.bzl#L177

and uniqify it, e.g.

load("@bazel_skylib//lib:collections.bzl", "collections")

...
dependencies = uniq(dependencies)

[thirtyseven]

Sorry about that, I think I accidentally deleted it while trying to minimize the repro case. Here's my requirements_mac.txt

[rickeylev]

Thanks.

Hm, my requirements_lock.txt (I'm on linux) is the same as your requirements_mac.txt.

I see you updated the repro files -- i tried those, still doesn't reproduce.

[thirtyseven]

I think it's a Mac specific issue, my guess is it's this line in ray[client]'s setup.py. I will try the dedup fix and if it works will submit a PR.

[thirtyseven]

I think the issue is a little more complicated, here is the BUILD file after macro expansion for me:

# /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_ray/BUILD.bazel:6:34
filegroup(
  name = "whl",
  visibility = ["//visibility:public"],
  generator_name = "ray-2.44.1-cp310-cp310-macosx_11_0_arm64.whl",
  generator_function = "whl_library_targets_from_requires",
  generator_location = "/private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_ray/BUILD.bazel:6:34",
  srcs = ["@@rules_python++pip+pypi_310_ray//:ray-2.44.1-cp310-cp310-macosx_11_0_arm64.whl"],
  data = ["@pypi//aiosignal:whl", "@pypi//click:whl", "@pypi//filelock:whl", "@pypi//frozenlist:whl", "@pypi//grpcio:whl", "@pypi//jsonschema:whl", "@pypi//msgpack:whl", "@pypi//packaging:whl", "@pypi//protobuf:whl", "@pypi//pyyaml:whl", "@pypi//requests:whl"] + select({"@@rules_python++pip+pypi_310_ray//:is_include_grpcio_true": ["@pypi//grpcio:whl"], "//conditions:default": []}),
)
# Rule whl instantiated at (most recent call last):
#   /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_ray/BUILD.bazel:6:34                  in <toplevel>
#   /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python+/python/private/pypi/whl_library_targets.bzl:81:24  in whl_library_targets_from_requires
#   /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python+/python/private/pypi/whl_library_targets.bzl:323:25 in whl_library_targets

In this case, grpcio seems to be both a regular and platform-specific dependency, maybe we can filter those out of dependencies_by_platform.
grpcio is being included in an extra both conditionally and unconditionally, and this isn't being handled correctly.

[thirtyseven]

Attempted to fix in #3513

[rickeylev]

I've put the fix for this in 1.8.0-rc4, which is going out now

[shayanhoshyari]

Ah, yep I also saw something like this with one of the google.cloud.... packages.