[Python] Uploading nightly wheels to the scientific python nightly channel

[jarrodmillman]

Describe the enhancement requested

It looks like you already build nightly Python wheels:

• https://github.com/apache/arrow/blob/main/docs/source/developers/python.rst#installing-nightly-packages

Would you be interested/willing to upload nightly wheels to he scientific-python nightly channel instead:

• https://anaconda.org/scientific-python-nightly-wheels/

This would make the wheels easier to discover by hosting them in a "standard" place. It will also make it easier for downstream projects to test against multiple packages, since they won't need to specify multiple locations in their CI scripts.

You can read more about why and how we are building / testing against nightly wheels here:
https://scientific-python.org/specs/spec-0004/

We also provide a GitHub action to upload and manage nightly builds:
https://github.com/scientific-python/upload-nightly-action/

We @matthewfeickert, @larsoner, @tupui are happy to help.

Component(s)

Python

Linked PRs

• GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to scientific python channel on Anaconda #43862
• GH-40216: [Python][CI][Packaging] Upload nightly wheels to main label of scientific-python-nightly-wheels channel #43932
• GH-40216: [Python][CI][Packaging] Don't upload sdist to scientific-python nightly channel (only wheels) #43943

[jorisvandenbossche]

cc @raulcd

This sounds as an interesting idea to me. My understanding is that our nightly wheels are not "official" release artifacts anyway, so it shouldn't matter too much where it is hosted? (also not sure if the current hosting location is actually tied to the Apache foundation)

I suppose the main cost is the cost of switching for current users of this (and for them to first find out that the old link no longer gives them the latest packages, at the time we would decide to stop uploading there as well)

[matthewfeickert]

👋 @jorisvandenbossche Just wanted to see if the team had any further thoughts on this. Let us know if we can help answer any questions.

[raulcd]

We probably could upload wheels to both places. In terms of discoverability we have to advise to users that the wheels provided are not official releases and should be used at their own risk. Similar to what we have at the documentation:
These packages are not official releases. Use them at your own risk.
This is because the Apache Release Policy (https://www.apache.org/legal/release-policy.html#what) says:

During the process of developing software and preparing a release, various packages are made available to the developer community for testing purposes. Do not include any links on the project website that might encourage non-developers to download and use nightly builds, snapshots, release candidates, or any other similar package. The only people who are supposed to know about such packages are the people following the dev list (or searching its archives) and thus aware of the conditions placed on the package. If you find that the general public are downloading such test packages, then remove them.

I think we are ok publishing the nightly wheels under the scientific python nightly channel.

[matthewfeickert]

I think we are ok publishing the nightly wheels under the scientific python nightly channel.

@raulcd Great! To move forward we need someone in the Arrow organization to do the following:

• Create an account on Anaconda Cloud
• Reply here with the link to your account (https://anaconda.org/ktou/)
• Make a new Anaconda Cloud API token with permissions described in https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects

• Save the API token as a repo GitHub Actions secret named ANACONDA_ORG_UPLOAD_TOKEN
• Direct me (@matthewfeickert) to a pyarrow nightly wheel for me to upload to create the package in the registry
• Verify that I have added you to the pyarrow team "Group" on https://anaconda.org/scientific-python-nightly-wheels (I've already added @jorisvandenbossche)
• Verify that I have added the package scientific-python-nightly-wheels/pyarrow to the pyarrow team "Group"

If you'd also like to use the scientific/upload-nightly-action GitHub Action please check out the action repo's README. If you need to collect wheels from different CI jobs, consider creating a new GitHub Actions workflow with logic similar to matplotlib's or Awkward's.

[kou]

Could you add me https://anaconda.org/ktou/ to the pyarrow team "Group" too?

[matthewfeickert]

@kou done. As soon as you generate an API token you can start uploading to https://anaconda.org/scientific-python-nightly-wheels/pyarrow. 👍 Let's leave this Issue open though until the first automated CI/CD upload passes.

[kou]

Thanks! I confirmed that I'm a member of the group.

@assignUser We need to ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN secret to our GitHub Actions. Could you also join the group, generate your API token and ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN?

[matthewfeickert]

Could you also join the group, generate your API token and ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN?

@assignUser Can you also share your Anaconda Cloud account name so I can add you to the pyarrow group?

[assignUser]

@matthewfeickert Sorry I missed the notifications 🙈 https://anaconda.org/assignuser

[matthewfeickert]

@assignUser added. 👍

[assignUser]

I'll take care of adding the token with infra