Using pip_data_exclude leads to errors

[det]

🐞 bug report

Affected Rule

pip_parse and pip_install

Is this a regression?

I don't know

Description

Using pip_data_exclude in pip_parse or pip_install leads to errors.

🔬 Minimal Reproduction

Add pip_data_exclude = ["test"], to pip_parse or pip_install.

🔥 Exception or Error

ERROR: no such package '@pip//': rules_python failed:  (Traceback (most recent call last):
  File "/home/chris/.cache/bazel/_bazel_chris/7d724a7e658b8a8712169d5f9ce79095/external/python3/python3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/chris/.cache/bazel/_bazel_chris/7d724a7e658b8a8712169d5f9ce79095/external/python3/python3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/chris/.cache/bazel/_bazel_chris/7d724a7e658b8a8712169d5f9ce79095/external/rules_python/python/pip_install/parse_requirements_to_bzl/__main__.py", line 5, in <module>
    main()
  File "/home/chris/.cache/bazel/_bazel_chris/7d724a7e658b8a8712169d5f9ce79095/external/rules_python/python/pip_install/parse_requirements_to_bzl/__init__.py", line 131, in main
    generate_parsed_requirements_contents(args)
  File "/home/chris/.cache/bazel/_bazel_chris/7d724a7e658b8a8712169d5f9ce79095/external/rules_python/python/pip_install/parse_requirements_to_bzl/__init__.py", line 51, in generate_parsed_requirements_contents
    args = deserialize_structured_args(args)
  File "/home/chris/.cache/bazel/_bazel_chris/7d724a7e658b8a8712169d5f9ce79095/external/rules_python/python/pip_install/parse_requirements_to_bzl/__init__.py", line 37, in deserialize_structured_args
    args[arg_name] = json.loads(args[arg_name])["args"]
KeyError: 'args'
)

🌍 Your Environment

Operating System:

  
Linux
  

Output of bazel version:

  
Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 07:33:24 2021 (1611214404)
Build timestamp: 1611214404
Build timestamp as int: 1611214404

  

Rules_python version:

  
0.2.0
  

Anything else relevant?

I am trying to exclude pyc files as pyyaml is erroneously including pyc in its wheel which leads to errors in bazel where it tries to overwrite a pyc file when creating the runfiles. pip_data_exclude seems like it should handle this except for the errors. I am also unable to even patch the rules_python archive as it seems the BUILD.bazel for the pip repos happens before the patch is applied.

Maybe .pyc should just be added the list of files excluded by default?

  
diff --git a/packaging/whl.py b/packaging/whl.py
index f30575a..f969cc2 100644
--- a/packaging/whl.py
+++ b/packaging/whl.py
@@ -185,7 +185,7 @@ load("{requirements}", "requirement")
 py_library(
     name = "pkg",
     srcs = glob(["**/*.py"], allow_empty = True),
-    data = glob(["**/*"], exclude=["**/*.py", "**/* *", "BUILD", "WORKSPACE"]),
+    data = glob(["**/*"], exclude=["**/*.py", "**/*.pyc", "**/* *", "BUILD", "WORKSPACE"]),
     # This makes this directory a top-level in the python import
     # search path for anything that depends on this.
     imports = ["."],
diff --git a/python/pip_install/extract_wheels/lib/bazel.py b/python/pip_install/extract_wheels/lib/bazel.py
index 95e8553..e3e188a 100644
--- a/python/pip_install/extract_wheels/lib/bazel.py
+++ b/python/pip_install/extract_wheels/lib/bazel.py
@@ -29,7 +29,7 @@ def generate_build_file_contents(
     there may be no Python sources whatsoever (e.g. packages written in Cython: like `pymssql`).
     """
 
-    data_exclude = ["*.whl", "**/*.py", "**/* *", "BUILD.bazel", "WORKSPACE"] + pip_data_exclude
+    data_exclude = ["*.whl", "**/*.py", "**/*.pyc", "**/* *", "BUILD.bazel", "WORKSPACE"] + pip_data_exclude
 
     return textwrap.dedent(
         """\
diff --git a/python/pip_install/repositories.bzl b/python/pip_install/repositories.bzl
index db9cd27..c59eee2 100644
--- a/python/pip_install/repositories.bzl
+++ b/python/pip_install/repositories.bzl
@@ -34,7 +34,7 @@ load("@rules_python//python:defs.bzl", "py_library")
 py_library(
     name = "lib",
     srcs = glob(["**/*.py"]),
-    data = glob(["**/*"], exclude=["**/*.py", "**/* *", "BUILD", "WORKSPACE"]),
+    data = glob(["**/*"], exclude=["**/*.py", "**/*.pyc", "**/* *", "BUILD", "WORKSPACE"]),
     # This makes this directory a top-level in the python import
     # search path for anything that depends on this.
     imports = ["."],

  

[thundergolfer]

Thanks for the bug report. I can't see the cause on first pass but shouldn't be too hard to sort this out. Are you sure this affects pip_install? I just tried to repro on a simple workspace and bazel build //... succeeded. I did the following:

pip_install(
    name = "pypi",
    requirements = "//helloworld:requirements.txt",
+   pip_data_exclude = ["test"],
 )

Maybe .pyc should just be added the list of files excluded by default?

Quite possibly

[hrfuller]

Its possible this is a bug in the serialization / deserialization of args in pip_parse.