Add py_wheel_push rule and automatic versioning capability for wheels #451

mvukov · 2021-04-09T14:57:36Z

PR Checklist

Please check if your PR fulfills the following requirements:

Does not include precompiled binaries, eg. .par files. See CONTRIBUTING.md for info
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

Bugfix
Feature (please, look at the "Scope of the project" section in the README.md file)
Code style update (formatting, local variables)
Refactoring (no functional changes, no api changes)
Build related changes
CI related changes
Documentation content changes
Other... Please describe:

What is the current behavior?

At the moment a wheel version can only be entered manually. There is no rule at the moment for pushing wheels to repositories.

What is the new behavior?

This PR adds capability to do automatic versioning/stamping as e.g.:

py_wheel(
    name = "my_wheel",
    distribution = "my_wheel",
    version = "{MY_WHEEL_VERSION}",
    deps = [":my_lib"],
)

where MY_WHEEL_VERSION can be calculated via workspace status by using --workspace_status_command included through the use of .bazelrc. For instance, we can calculate MY_WHEEL_VERSION using some git info. This is handy for automating pushes of wheels on CI.

The second feature this PR introduces is a new py_wheel_push rule that internally uses twine. This enables pushing wheels to private wheel repositories.

Does this PR introduce a breaking change?

Yes
No

The new py_wheel outputs a wheel named <target_name>.whl and a version file <target_name>_version.txt. The old/current rule outputs the wheel with (already) compliant name. In principle, the version is only relevant when pushing wheels and the new rule for pushing wheels uses the version file to assemble the final compliant wheel file name.

Other information

Docs/testing are still missing, but would like first we agree on the approach.

google-cla · 2021-04-09T14:58:03Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-09T15:00:52Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-09T15:01:55Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-09T15:13:05Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

pstradomski

Can we split the versioning change from the upload one?

Regarding approach: wouldn't it be much simpler to just use https://docs.bazel.build/versions/master/be/make-variables.html support? Then you'd just need to call ctx.expand_make_variables on ctx.attr.version

pstradomski · 2021-04-09T15:55:40Z

python/packaging.bzl

+        progress_message = "Versioning wheel",
+    )
+
+    wheel_file = ctx.actions.declare_file(ctx.label.name + ".whl")


Isn't this generating a wrong filename though, without all the platform and version information?

I think if I make a wheel, then my intention would be to push it to a repo. At least this is my current use-case. In this light, name of a wheel file that py_wheel outputs is less relevant IMO. py_wheel_push assembles a correct file name before pushing a wheel. But I may be missing some other important use-cases. I am open to discuss this further.

mvukov · 2021-04-12T08:57:00Z

Regarding approach: wouldn't it be much simpler to just use https://docs.bazel.build/versions/master/be/make-variables.html support? Then you'd just need to call ctx.expand_make_variables on ctx.attr.version

What I have in mind is the following. Let's say that CI is set up to push a wheel to a private repo on a tag. I want to use that git tag as a version for a wheel -- let's assume for simplicity the git tag is conformant according to PEP440. After doing some research, I found that only way to inject this info to Bazel is via a workspace status script. Now, the contents of ctx.info_file and ctx.version_file cannot be used in a Starlark script -- please correct me if I am wrong. That's why I made wheelversioner.py that figures out a version and dumps that info to a file.

Please paste here a small snippet with make variables, in case you have a clear idea in mind. Thanks in advance.

I am not sure how to split the PR, the versioning PR would not be standalone without the new py_wheel_push rule... WDYT?

pstradomski · 2021-04-12T11:45:27Z

Well, all other tools (pip in particular) require the wheel file to have the correct name, so breaking that is a big no. pon., 12 kwi 2021 o 10:57 Milan Vukov ***@***.***> napisał(a):

…

Regarding approach: wouldn't it be much simpler to just use https://docs.bazel.build/versions/master/be/make-variables.html support? Then you'd just need to call ctx.expand_make_variables on ctx.attr.version What I have in mind is the following. Let's say that CI is set up to push a wheel to a private repo on a tag. I want to use that git tag as a version for a wheel -- let's assume for simplicity the git tag is conformant according to PEP440. After doing some research, I found that only way to inject this info to Bazel is via a workspace status script. Now, the contents of ctx.info_file and ctx.version_file cannot be used in a Starlark script -- please correct me if I am wrong. That's why I made wheelversioner.py that figures out a version and dumps that info to a file. Please paste here a small snippet with make variables, in case you have a clear idea in mind. Thanks in advance. I am not sure how to split the PR, the versioning PR would not be standalone without the new py_wheel_push rule... WDYT? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#451 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKU4J4K2SUT4WLTWFGACZVDTIKYXBANCNFSM42VEIAHQ> .

-- Paweł Stradomski

pstradomski · 2021-04-12T11:57:23Z

Wrt getting version info, you should be able to just re-export the git information into TemplateVariableInfo. See here for defining a variable: https://github.com/bazelbuild/examples/blob/master/make-variables/testapp/defs.bzl#L4 See here for reading version info from a build rule: https://github.com/JulianSchmid/example-bazel-add-git-hash/blob/master/bake_git_hash.bzl pon., 12 kwi 2021 o 13:44 Paweł Stradomski ***@***.***> napisał(a):

…

Well, all other tools (pip in particular) require the wheel file to have the correct name, so breaking that is a big no. pon., 12 kwi 2021 o 10:57 Milan Vukov ***@***.***> napisał(a): > Regarding approach: wouldn't it be much simpler to just use > https://docs.bazel.build/versions/master/be/make-variables.html support? > Then you'd just need to call ctx.expand_make_variables on ctx.attr.version > > What I have in mind is the following. Let's say that CI is set up to push > a wheel to a private repo on a tag. I want to use that git tag as a version > for a wheel -- let's assume for simplicity the git tag is conformant > according to PEP440. After doing some research, I found that only way to > inject this info to Bazel is via a workspace status script. Now, the > contents of ctx.info_file and ctx.version_file cannot be used in a > Starlark script -- please correct me if I am wrong. That's why I made > wheelversioner.py that figures out a version and dumps that info to a > file. > > Please paste here a small snippet with make variables, in case you have a > clear idea in mind. Thanks in advance. > > I am not sure how to split the PR, the versioning PR would not be > standalone without the new py_wheel_push rule... WDYT? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#451 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AKU4J4K2SUT4WLTWFGACZVDTIKYXBANCNFSM42VEIAHQ> > . > -- Paweł Stradomski

-- Paweł Stradomski

pstradomski · 2021-04-12T12:01:30Z

Btw your PR already populates a provider with the desired version number, so you can just as well populate TemplateVariableInfo with that desired version number. pon., 12 kwi 2021 o 13:56 Paweł Stradomski ***@***.***> napisał(a):

…

Wrt getting version info, you should be able to just re-export the git information into TemplateVariableInfo. See here for defining a variable: https://github.com/bazelbuild/examples/blob/master/make-variables/testapp/defs.bzl#L4 See here for reading version info from a build rule: https://github.com/JulianSchmid/example-bazel-add-git-hash/blob/master/bake_git_hash.bzl pon., 12 kwi 2021 o 13:44 Paweł Stradomski ***@***.***> napisał(a): > Well, all other tools (pip in particular) require the wheel file to have > the correct name, so breaking that is a big no. > > > > pon., 12 kwi 2021 o 10:57 Milan Vukov ***@***.***> > napisał(a): > >> Regarding approach: wouldn't it be much simpler to just use >> https://docs.bazel.build/versions/master/be/make-variables.html >> support? Then you'd just need to call ctx.expand_make_variables on >> ctx.attr.version >> >> What I have in mind is the following. Let's say that CI is set up to >> push a wheel to a private repo on a tag. I want to use that git tag as a >> version for a wheel -- let's assume for simplicity the git tag is >> conformant according to PEP440. After doing some research, I found that >> only way to inject this info to Bazel is via a workspace status script. >> Now, the contents of ctx.info_file and ctx.version_file cannot be used >> in a Starlark script -- please correct me if I am wrong. That's why I made >> wheelversioner.py that figures out a version and dumps that info to a >> file. >> >> Please paste here a small snippet with make variables, in case you have >> a clear idea in mind. Thanks in advance. >> >> I am not sure how to split the PR, the versioning PR would not be >> standalone without the new py_wheel_push rule... WDYT? >> >> — >> You are receiving this because you commented. >> Reply to this email directly, view it on GitHub >> <#451 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AKU4J4K2SUT4WLTWFGACZVDTIKYXBANCNFSM42VEIAHQ> >> . >> > > > -- > Paweł Stradomski > -- Paweł Stradomski

-- Paweł Stradomski

mvukov · 2021-04-13T07:52:34Z

Btw your PR already populates a provider with the desired version number,
so you can just as well populate TemplateVariableInfo with that desired
version number.

In WheelInfo I have version_file field, which routes the file with version generated in py_wheel rule using wheelversioner.py. wheelpusher.py then reads this file to get the version and assembles correct file name for the wheel.

Unless I am mistaken, Bazel context doesn't provide functionality to read data from files. If I would be able to read a file using ctx, then I could assign it to an output variable or use TemplateVariableInfo.

Thanks for the examples. https://github.com/JulianSchmid/example-bazel-add-git-hash/blob/master/bake_git_hash.bzl dumps git info to a file, similar to how I dump wheel version to a file.

The bottom line is, as I see it, py_wheel info needs to declare the wheel file and return it via a provider such that py_wheel_push can consume it. To declare the wheel file I need to know it's file name. The file name depends on the version which I cannot fetch from a file when the script is processed, since it could depend ctx.info_file and ctx.version_file. So, py_wheel sets a "dummy" name <target_name>.whl in py_wheel. py_wheel_push, wheelpusher.py in fact, is in charge to assemble correct file name for the wheel before pushing it to a repo.

pstradomski · 2021-04-13T10:08:35Z

ok, that's a bummer. You could try --define flag to bazel as another way to pass git info, though it's not as nice as workspace info. Anyway, whatever approach, the wheel file must have the correct name, otherwise you'd break existing usecases. wt., 13 kwi 2021 o 09:52 Milan Vukov ***@***.***> napisał(a):

…

Btw your PR already populates a provider with the desired version number, so you can just as well populate TemplateVariableInfo with that desired version number. In WheelInfo I have version_file field, which routes the file with version generated in py_wheel rule using wheelversioner.py. wheelpusher.py then reads this file to get the version and assembles correct file name for the wheel. Unless I am mistaken, Bazel context doesn't provide functionality to read data from files. If I would be able to read a file using ctx, then I could assign it to an output variable or use TemplateVariableInfo. Thanks for the examples. https://github.com/JulianSchmid/example-bazel-add-git-hash/blob/master/bake_git_hash.bzl dumps git info to a file, similar to how I dump wheel version to a file. The bottom line is, as I see it, py_wheel info needs to declare the wheel file and return it via a provider such that py_wheel_push can consume it. To declare the wheel file I need to know it's file name. The file name depends on the version which I cannot fetch from a file when the script is processed, since it could depend ctx.info_file and ctx.version_file. So, py_wheel sets a "dummy" name <target_name>.whl in py_wheel. py_wheel_push, wheelpusher.py in fact, is in charge to assemble correct file name for the wheel before pushing it to a repo. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#451 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKU4J4OHSKKOMKDW7PIOGMLTIPZ5BANCNFSM42VEIAHQ> .

-- Paweł Stradomski

mvukov · 2021-04-13T16:27:51Z

How about the following: if the user specifies version = "FOO" we immediately output correct wheel file name from py_wheel. On the other hand, if the user gives version = "{FOO}", py_wheel sets the wheel name <target_name>.whl. py_wheel_push in that case would always compute the correct file name before pushing to a repo. WDYT?

pstradomski · 2021-04-14T08:30:25Z

I'd suggest checking first in the main bazel issue tracker if there are any plans to make version info (or sth similar) available to rule implementations. Currently only --define comes to my mind, I'm not sure how difficult it would be for you to use. wt., 13 kwi 2021 o 18:28 Milan Vukov ***@***.***> napisał(a):

…

How about the following: if the user specifies version = "FOO" we immediately output correct wheel file name from py_wheel. On the other hand, if the user gives version = "{FOO}", py_wheel sets the wheel name <target_name>.whl. py_wheel_push in that case would always compute the correct file name before pushing to a repo. WDYT? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#451 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKU4J4P53Z2BTNHDVRAPS7TTIRWJVANCNFSM42VEIAHQ> .

-- Paweł Stradomski

mvukov · 2021-04-15T14:41:04Z

@googlebot I signed it!

google-cla · 2021-04-15T14:41:13Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

franklouwers · 2021-04-22T06:40:10Z

@googlebot we signed it

github-actions · 2021-12-31T22:50:35Z

This Pull Request has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

github-actions · 2022-01-30T22:50:37Z

This PR was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

mvukov requested review from andyscott, brandjon, lberki and thundergolfer as code owners April 9, 2021 14:57

google-cla bot added the cla: no label Apr 9, 2021

Add py_wheel_push rule and automatic versioning capability for wheels

1840fef

mvukov force-pushed the feature/wheel_push branch from f2acfab to 1840fef Compare April 9, 2021 15:00

rm unused import

a24add6

mvukov added 2 commits April 9, 2021 17:12

Update internal deps

298fc84

Make linter happy

691b189

pstradomski requested changes Apr 9, 2021

View reviewed changes

mvukov mentioned this pull request Apr 15, 2021

Use of ctx.info_file and ctx.version_file in rule implementations bazelbuild/bazel#13360

Open

google-cla bot added cla: yes and removed cla: no labels Apr 22, 2021

mvukov mentioned this pull request Aug 30, 2021

py_wheel: support version_file #516

Closed

github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Dec 31, 2021

github-actions bot closed this Jan 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add py_wheel_push rule and automatic versioning capability for wheels #451

Add py_wheel_push rule and automatic versioning capability for wheels #451

mvukov commented Apr 9, 2021

google-cla bot commented Apr 9, 2021

google-cla bot commented Apr 9, 2021

google-cla bot commented Apr 9, 2021

google-cla bot commented Apr 9, 2021

pstradomski left a comment

pstradomski Apr 9, 2021

mvukov Apr 12, 2021 •

edited

Loading

mvukov commented Apr 12, 2021

pstradomski commented Apr 12, 2021 via email

pstradomski commented Apr 12, 2021 via email

pstradomski commented Apr 12, 2021 via email

mvukov commented Apr 13, 2021

pstradomski commented Apr 13, 2021 via email

mvukov commented Apr 13, 2021

pstradomski commented Apr 14, 2021 via email

mvukov commented Apr 15, 2021

google-cla bot commented Apr 15, 2021

franklouwers commented Apr 22, 2021

github-actions bot commented Dec 31, 2021

github-actions bot commented Jan 30, 2022

Add py_wheel_push rule and automatic versioning capability for wheels #451

Add py_wheel_push rule and automatic versioning capability for wheels #451

Conversation

mvukov commented Apr 9, 2021

PR Checklist

PR Type

What is the current behavior?

What is the new behavior?

Does this PR introduce a breaking change?

Other information

google-cla bot commented Apr 9, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

google-cla bot commented Apr 9, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

google-cla bot commented Apr 9, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

google-cla bot commented Apr 9, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

pstradomski left a comment

Choose a reason for hiding this comment

pstradomski Apr 9, 2021

Choose a reason for hiding this comment

mvukov Apr 12, 2021 • edited Loading

Choose a reason for hiding this comment

mvukov commented Apr 12, 2021

pstradomski commented Apr 12, 2021 via email

pstradomski commented Apr 12, 2021 via email

pstradomski commented Apr 12, 2021 via email

mvukov commented Apr 13, 2021

pstradomski commented Apr 13, 2021 via email

mvukov commented Apr 13, 2021

pstradomski commented Apr 14, 2021 via email

mvukov commented Apr 15, 2021

google-cla bot commented Apr 15, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

franklouwers commented Apr 22, 2021

github-actions bot commented Dec 31, 2021

github-actions bot commented Jan 30, 2022

mvukov Apr 12, 2021 •

edited

Loading