Switch Python rules to use toolchains

[brandjon]

This bug tracks designing, implementing, and migrating built-in Python rules to use the toolchain mechanism for resolving the Python runtime. This replaces the legacy mechanism of specifying --python_top to globally make all Python targets look at a specific py_runtime, or omitting --python_top to use the system default "python" command.

This is prioritized. Design doc to follow.

[brandjon]

(Plan is to use this work to also address #4815, rather than implement an ad hoc solution to that bug.)

[brandjon]

Design doc here.

Discussion can proceed in this thread. Initial reviewers are @katre (Bazel, Toolchains), @mrovner (Python), and @nlopezgi (Remote Execution).

[katre]

Overall, this looks good. A few points:

1. Get rid of the host_python_toolchain, just use the autodetecting version. I don't think the time spent running the autodetection script will really hurt python performance, will it?
2. In the Backwards Compatbility section, you mention deleting several flags: I do not think flags can be conditionally deleted based on another flag. Ignoring the values should be sufficient, and they can be deleted in a post-migration cleanup.

[brandjon]

1. Ok, I'll remove host_python_toolchain. I don't have a strong opinion about its impact on performance, but it's worth remembering that it will still be superseded by any other applicable toolchain that the user registers.

2. By "delete" I really mean "fail-fast if the user tries to specify them while the incompatible flag is enabled". This can be implemented by checking the flag value for null, and logging an error in the fragment's reportInvalidOptions() (see example). The true deletion would happen around the time the old code paths and incompatible flag are removed.

[katre]

I think the distinction between "delete" and "ignore with a warning" is worth calling out.

[brandjon]

(See also Nicolas's comments here.)

[brandjon]

Replying to @nlopezgi's concern about toolchains and extensibility here:

I think you need something similar as there are rules that execute the native py_binary (e.g., rules_docker does here: https://github.com/bazelbuild/rules_docker/blob/master/python/image.bzl#L84) and there might be something special that these type of users need to do to work with the new py toolchains properly.

Users who operate at the level of BUILD targets and macros only need to worry about having definitions for their platforms and python toolchains, i.e. the py_runtime_pair target and an associated toolchain target referencing it, with appropriate constraints. Whatever the build graph looks like, it should Just Work.

Users who are writing new rules that interoperate with the Python rules may need to manipulate Python toolchains. To receive a Python toolchain, they should declare @bazel_tools//tools/python:toolchain_type as a required toolchain type in their rule and receive it with ctx.toolchains[<same label>]. This would give them back the ToolchainInfo with the schema described in the doc. This would be necessary for instance to write a replica of py_binary in Starlark. Conversely, users can write a replacement for py_runtime or py_runtime_pair by simply returning an appropriate PyRuntimeInfo / ToolchainInfo provider.

I didn't go into too much detail about how to write rules that produce or consume Python toolchains (aside from a description of the provider schema) because this is already covered by the general toolchain documentation linked in the doc's abstract.

[dslomov]

(please do not assign issues to more than one team-*)