Publishing nightly packages

[jakirkham]

When doing active development, it can be quite useful to have nightly packages of things that are being developed in tandem. In the context of Dask, this comes up when developing a downstream library that depends on Dask (often requiring the very latest changes). Am curious what others think about supplying nightly packages of dask and distributed?

[kkraus14]

FYI: This is an ongoing problem for us at cuDF as there's often PRs of ours where we need to make matching changes in both cuDF and Dask and we end up leaving things in a broken state until a new Dask release is pushed out. If there were nightly conda and pypi packages that would largely solve our problems.

[mrocklin]

I personally have no objections to this. Where would the packages live? A new Dask channel? Does conda-forge have a dev channel? Who would maintain this and keep it running (I'm guessing @jakirkham and the NVIDIA folks). Are there any other concerns that we should be aware of? John, you're probably the expert here.

…

On Mon, Jul 27, 2020 at 1:21 PM Keith Kraus ***@***.***> wrote: FYI: This is an ongoing problem for us at cuDF as there's often PRs of ours where we need to make matching changes in both cuDF and Dask and we end up leaving things in a broken state until a new Dask release is pushed out. If there were nightly conda and pypi packages that would largely solve our problems. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGOLC7CCRTNDXMWIADR5XOWJANCNFSM4PJCDGLA> .

[mrocklin]

I think that most people will pip install from master when they want an up-to-date version. I'm guessing that that doesn't work for RAPIDS development for some reason though.

…

On Mon, Jul 27, 2020 at 1:24 PM Matthew Rocklin ***@***.***> wrote: I personally have no objections to this. Where would the packages live? A new Dask channel? Does conda-forge have a dev channel? Who would maintain this and keep it running (I'm guessing @jakirkham and the NVIDIA folks). Are there any other concerns that we should be aware of? John, you're probably the expert here. On Mon, Jul 27, 2020 at 1:21 PM Keith Kraus ***@***.***> wrote: > FYI: This is an ongoing problem for us at cuDF as there's often PRs of > ours where we need to make matching changes in both cuDF and Dask and we > end up leaving things in a broken state until a new Dask release is pushed > out. If there were nightly conda and pypi packages that would largely solve > our problems. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#76 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACKZTGOLC7CCRTNDXMWIADR5XOWJANCNFSM4PJCDGLA> > . >

[kkraus14]

I think that most people will pip install from master when they want an up-to-date version. I'm guessing that that doesn't work for RAPIDS development for some reason though.
…
On Mon, Jul 27, 2020 at 1:24 PM Matthew Rocklin @.> wrote: I personally have no objections to this. Where would the packages live? A new Dask channel? Does conda-forge have a dev channel? Who would maintain this and keep it running (I'm guessing @jakirkham and the NVIDIA folks). Are there any other concerns that we should be aware of? John, you're probably the expert here. On Mon, Jul 27, 2020 at 1:21 PM Keith Kraus @.> wrote: > FYI: This is an ongoing problem for us at cuDF as there's often PRs of > ours where we need to make matching changes in both cuDF and Dask and we > end up leaving things in a broken state until a new Dask release is pushed > out. If there were nightly conda and pypi packages that would largely solve > our problems. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#76 (comment)>, or > unsubscribe > https://github.com/notifications/unsubscribe-auth/AACKZTGOLC7CCRTNDXMWIADR5XOWJANCNFSM4PJCDGLA > . >

I think the challenge here is there's no clean and nice way to integrate that into a conda package dependency. I.E. as we make a coordinated change in Dask and RAPIDS, once the PR is merged in Dask we could upgrade the conda pinning in the RAPIDS recipe to ensure we get the newer Dask version.

[quasiben]

One option for nightlies is that we could publish to the dask channel on anaconda:
https://anaconda.org/dask/

[jakirkham]

I personally have no objections to this.

Great!

Where would the packages live? A new Dask channel? Does conda-forge have a dev channel?

A dask or dask-nightly channel would make sense. conda-forge doesn't support nightlies.

Who would maintain this and keep it running (I'm guessing @jakirkham and the NVIDIA folks).

That's my guess as well. Though it would be nice to have someone from the cuDF side on the hook as well (since that's driving the demand 😉).

Are there any other concerns that we should be aware of? John, you're probably the expert here.

The main thing to consider would be where these get built, how to signal these are dev packages, and what to do regarding communicating build issues to relevant maintainers.

As far as building goes, doing this as part of Dask and Distributed CI seems like the natural place. This ensures when new changes get integrated packages are published in pretty rapid order. It also avoids needing to add a separate thing to maintain and coordinate between.

The next thing would be communicating to users that these are only dev packages and they should go to more formal channels (conda-forge, defaults) for final releases. This may involve naming the channel accordingly or using a label to indicate these are development builds. A small blurb in the docs next to other install instructions might also be useful for relaying this information.

Lastly it would be good to setup a GitHub team for those maintaining nightlies to direct communication/issues in that direction.

Those are the other additional points I can think of atm. Though please feel free to raise others 🙂

[kkraus14]

That's my guess as well. Though it would be nice to have someone from the cuDF side on the hook as well (since that's driving the demand 😉).

FWIW it's more than just the cuDF side, cuML runs into these situations as well, albeit not as frequently as cuDF. I'm happy to be "on the hook".

The main thing to consider would be where these get built, how to signal these are dev packages, and what to do regarding communicating build issues to relevant maintainers.

As far as building goes, doing this as part of Dask and Distributed CI seems like the natural place. This ensures when new changes get integrated packages are published in pretty rapid order. It also avoids needing to add a separate thing to maintain and coordinate between.

Agreed. Looks like there's Travis CI jobs that run after PRs are merged that could likely be a good place for uploads to take place given the packages are pure python: https://travis-ci.org/github/dask/dask/builds

[jakirkham]

Thanks Keith! Happy to have someone from cuML as well 🙂

[quasiben]

If we want to publish to https://anaconda.org/dask/ I have admin access and can generate tokens. @kkraus14 is there someone who can sit with John and I while we get this up and running

[kkraus14]

If we want to publish to https://anaconda.org/dask/ I have admin access and can generate tokens. @kkraus14 is there someone who can sit with John and I while we get this up and running

That someone is me 😄

[jsignell]

This would be on the dev label of the dask channel right?

[TomAugspurger]

Part of the uploading job should probably delete older versions. The ones uploaded to https://anaconda.org/scipy-wheels-nightly/ are cleared after some number of days.

[charlesbluca]

Reviving this issue because there has recently been some demand from RAPIDS folk to publish nightlies of dask-sql:

https://github.com/dask-contrib/dask-sql

and the dask channel seems like a sensible place to host these. Would there be any objection to using dask-sql as a trial for publishing nightlies to dask, and then pushing on Dask/Distributed nightlies once that's working?