Python Zip Generated With --build_python_zip Do Not Have All of the Runfiles

[burak-db]

Description of the problem / feature request:

Question is about python builds specifically about --build_python_zip flag. We have a custom rule that emits PyInfo provider so we are able to depend on our custom rule from py_library and py_binary rules. Our custom rule basically unzips a wheel package and adds it to the imports field of PyInfo so that it is added to the python path when executing the binary. This works well with bazel run //path_to_py_binary but when we use --build_python_zip flag we see that our custom rules were not executed and the contents of the wheel packages were not unzipped. --build_python_zip flag changing the behaviour of py_binary which seems like an issue to me.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Create a custom rule that unzips a wheel and adds it to the PyInfo(imports) and then zip generated with --build_python_zip does not include those unzipped wheel packages whereas running py_binary with bazel run does include

What operating system are you running Bazel on?

mac os x

What's the output of bazel info release?

release 4.2.1

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Replace this line with your answer.

Have you found anything relevant by searching the web?

I've seen some other issues with build_python_zip flag but was not very similar to this one.

Any other information, logs, or outputs that you want to share?

[burak-db]

@aiuto what do you think about this issue?

[aiuto]

I don't have an opinion. I was just doing incoming triage. This is a question for the python rules folks.

[burak-db]

okay I see. Shall we ping any of them?

[burak-db]

cc @meteorcloudy since you implemented the flag in the first place. we have the following rule. which we are using as a dependency to rules_python. it works except when we pass --build_python_zip. with --build_python_zip the directories we create in the below rule becomes empty(in the output zip file)

def _python_unwheel_impl(ctx):
    output = ctx.actions.declare_directory(ctx.attr.name)

    wheels = ctx.files.wheels
    commands = [
        "unzip -d {} {}".format(output.path, wheel.path)
        for wheel in wheels
    ]

    ctx.actions.run_shell(
        outputs = [output],
        inputs = wheels,
        mnemonic = "Unwheel",
        progress_message = "Unwheel: unzipping {} wheels".format(len(wheels)),
        command = "; ".join(commands),
        execution_requirements = {"no-cache": "", "no-remote": ""},
    )

    return [
        PyInfo(
            imports = depset(order = "default", direct = [paths.join(ctx.workspace_name, output.short_path)], transitive = []),
            transitive_sources = depset(order = "postorder", direct = [output], transitive = []),
        ),
        DefaultInfo(
            files = depset(direct = [output]),
            runfiles = ctx.runfiles(files = [output]),
        ),
    ]

[meteorcloudy]

@burak-db Sorry for the late reply here. The code for creating the python zip archive is here It should contain all runfiles of the binary.
Can you help provide a minimal reproduce case so that I can debug it? Thanks!

[AlessandroPatti]

@meteorcloudy I am able to reproduce the issue with the following minimal example:

$ touch WORKSPACE
$ echo 'import lib.one, lib.two; print("Ok")' > main.py
$ echo 'genrule(name = "tree", outs = ["lib"], cmd = "mkdir $@; touch $@/one.py $@/two.py")' > BUILD
$ echo 'py_binary(name  = "bin", srcs = ["main.py"], data = ["lib"], main = "main.py")' >> BUILD
$ bazel run //:bin --ui_event_filters=ERROR --noshow_progress
Ok
$ bazel run //:bin --ui_event_filters=ERROR --noshow_progress --build_python_zip
Traceback (most recent call last):
  File "/var/folders/m7/r8g2mz816t9fn7gytrj2f0300000gp/T/Bazel.runfiles_koAc6Y/runfiles/__main__/main.py", line 1, in <module>
    import lib.one, lib.two; print("Ok")
ModuleNotFoundError: No module named 'lib.one'
$ unzip -t bazel-bin/bin.zip
Archive:  bazel-bin/bin.zip
    testing: __main__.py              OK
    testing: __init__.py              OK
    testing: runfiles/__main__/__init__.py   OK
    testing: runfiles/__main__/main.py   OK
    testing: runfiles/__main__/lib/   OK
    testing: runfiles/bazel_tools/tools/python/py3wrapper.sh   OK
No errors detected in compressed data of bazel-bin/bin.zip.

Looks like the folder is added but not with its content.

[meteorcloudy]

@AlessandroPatti That is work as intended. The genrule only declares "lib" as an output, which is a directory, Bazel won't recursively collect files under it. Therefore, normally outs should specify a list of files instead of a directory. Also "lib" should be listed a dependency of "bin" through "deps" attribute instead of "data".

[AlessandroPatti]

@meteorcloudy would it make sense to have a flag like --python_zip_expand_directory? Starlark actions can already expand directories using Args, so there is a precendent already, and it is very confusing to have a python_binary resulting in two different behaviours dependeing on whether zipping is enabled or not.