Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use hermetic python #240

Merged
merged 3 commits into from
Dec 14, 2023
Merged

Use hermetic python #240

merged 3 commits into from
Dec 14, 2023

Conversation

lpawelcz
Copy link
Collaborator

@lpawelcz lpawelcz commented Dec 8, 2023

Fixes #209

This PR removes cpython toolchain and embedded_python_interpreter. It replaces those with python interpreter taken from rules_python which is a proper way of handling hermetic python toolchain.
Additional modifications take into account explicit calls to python in rules like cocotb or gds_write.
Those modifications are also required to allow the usage of gds_write rule in XLS repository (see google/xls#1031)

CC @proppy

@lpawelcz
Copy link
Collaborator Author

lpawelcz commented Dec 8, 2023

I'm not sure what is causing the CI failures and I don't have the access to the logs. @proppy, @QuantamHD could you please paste those for me?

@mithro
Copy link
Member

mithro commented Dec 14, 2023

Step #1:   /workspace/dependency_support/dependency_support.bzl:86:24: in dependency_support
Step #1:   /workspace/dependency_support/com_google_protobuf/com_google_protobuf.bzl:21:10: in com_google_protobuf
Step #1:   /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Step #1: Repository rule git_repository defined at:
Step #1:   /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
Step #1: Loading: 0 packages loaded
Step #1:     currently loading: tools
Step #1: INFO: Repository python39_x86_64-unknown-linux-gnu instantiated at:
Step #1:   /workspace/WORKSPACE:50:27: in <toplevel>
Step #1:   /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/rules_python/python/repositories.bzl:555:26: in python_register_toolchains
Step #1: Repository rule python_repository defined at:
Step #1:   /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/rules_python/python/repositories.bzl:384:36: in <toplevel>
Step #1: ERROR: An error occurred during the fetch of repository 'python39_x86_64-unknown-linux-gnu':
Step #1:    Traceback (most recent call last):
Step #1: 	File "/builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/rules_python/python/repositories.bzl", line 200, column 25, in _python_repository_impl
Step #1: 		fail("The current user is root, please run as non-root when using the hermetic Python interpreter. See https://github.com/bazelbuild/rules_python/pull/713.")
Step #1: Error in fail: The current user is root, please run as non-root when using the hermetic Python interpreter. See https://github.com/bazelbuild/rules_python/pull/713.
Step #1: ERROR: /workspace/WORKSPACE:50:27: fetching python_repository rule //external:python39_x86_64-unknown-linux-gnu: Traceback (most recent call last):
Step #1: 	File "/builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/rules_python/python/repositories.bzl", line 200, column 25, in _python_repository_impl
Step #1: 		fail("The current user is root, please run as non-root when using the hermetic Python interpreter. See https://github.com/bazelbuild/rules_python/pull/713.")
Step #1: Error in fail: The current user is root, please run as non-root when using the hermetic Python interpreter. See https://github.com/bazelbuild/rules_python/pull/713.
Step #1: ERROR: Skipping '//tools:report_status_to_github': error loading package 'tools': Encountered error while reading extension file 'requirements.bzl': no such package '@rules_hdl_pip_deps//': no such package '@python39_x86_64-unknown-linux-gnu//': The current user is root, please run as non-root when using the hermetic Python interpreter. See https://github.com/bazelbuild/rules_python/pull/713.
Step #1: WARNING: Target pattern parsing failed.
Step #1: ERROR: error loading package 'tools': Encountered error while reading extension file 'requirements.bzl': no such package '@rules_hdl_pip_deps//': no such package '@python39_x86_64-unknown-linux-gnu//': The current user is root, please run as non-root when using the hermetic Python interpreter. See https://github.com/bazelbuild/rules_python/pull/713.
Step #1: INFO: Elapsed time: 12.657s
Step #1: INFO: 0 processes.
Step #1: FAILED: Build did NOT complete successfully (0 packages loaded)
Step #1: ERROR: Build failed. Not running target
Step #1: FAILED: Build did NOT complete successfully (0 packages loaded)
Step #1: FAILED: Build did NOT complete successfully (0 packages loaded)
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/bazel" failed: step exited with non-zero status: 1

WORKSPACE Show resolved Hide resolved
@lpawelcz lpawelcz force-pushed the use-hermetic-python branch from f9c7107 to 17db21c Compare December 14, 2023 15:28
Remove cpython toolchain and embedded_python_interpreter.
Use provided hermetic python interpreter from rules_python instead

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>
@QuantamHD QuantamHD merged commit c053626 into hdl:main Dec 14, 2023
4 checks passed
@lpawelcz lpawelcz deleted the use-hermetic-python branch December 15, 2023 06:34
@@ -132,10 +132,12 @@ def _get_test_command(ctx, verilog_files, vhdl_files):
seed_args = " --seed {}".format(ctx.attr.seed) if ctx.attr.seed != "" else ""

test_module_args = _pymodules_to_argstring(ctx.files.test_module, "test_module")
python_interpreter = ctx.toolchains["@bazel_tools//tools/python:toolchain_type"].py3_runtime.interpreter.path
Copy link
Collaborator

@proppy proppy Dec 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we be using:

toolchain="@rules_python//python:current_py_toolchain"

and

${PYTHON3}

as recommended in https://rules-python.readthedocs.io/en/latest/getting-started.html#toolchain-usage-in-other-rules ?

@@ -81,6 +82,7 @@ def _gds_write_impl(ctx):
gds_allow_empty_args +
" --out {}".format(final_gds.path),
tools = depset([ctx.executable._gds_write]),
toolchain = "//python:current_py_toolchain",
Copy link
Collaborator

@proppy proppy Dec 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we be using:

toolchain="@rules_python//python:current_py_toolchain"

as recommended in https://rules-python.readthedocs.io/en/latest/getting-started.html#toolchain-usage-in-other-rules ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

use rules_python hermetic toolchains
5 participants