Draft release feature on main archive to allow testing a release before it goes live

[njsmith]

As per this discussion thread:

https://mail.python.org/pipermail/distutils-sig/2015-October/thread.html#27234

it would be very nice if there where better ergonomics around package uploads -- in particular some way to upload a new release, and then take a look over it to double-check that everything is correct before you -- as a second step -- hit the button to make it "go live". Glyph suggests that in particular he'd like to be able to actually run a test install against the uploaded data as an end-to-end test:

https://mail.python.org/pipermail/distutils-sig/2015-October/027259.html

which indeed sounds glorious, and I think the super-slick way to do this would be to provide a unique private index URL which gives a view of what pypi would look like after your release goes live, and could be used like

pip install --index-url https://pypi.python.org/tmp/SOMEPKG/acd1538afe267/ SOMEPKG

(https://mail.python.org/pipermail/distutils-sig/2015-October/027263.html)

The idea would be basically that any request to /tmp/SOMEPKG/acd1538afe267/WHATEVER would return the same thing as a request to /WHATEVER, except for requests that would be affected by the addition of the new release, would act as if the release had already been made.

The use of a unique URL for each trial upload means that this still plays nicely with caching/CDNs. The inclusion of the package name in the tmp URL allows people to double-check that if they see a URL like this, then they know that the files there were actually uploaded by someone who is trusted to upload that package. You'd want to expire them after some short-but-reasonable time (a few days?) to prevent them being abused as private indices by unscrupulous people, and also just for general hygiene, but that's fine.

Obviously this is very much a post-"become PyPI" wishlist priority request.

[njsmith]

Another advantage of this feature would be that it would provide a way for distributed teams to coordinate releases. E.g. one could use a workflow like:

• main release manager uploads (hopefully) final source release to one of these staging areas
• the volunteer who provides osx builds fetches the source from this staging area, builds wheels, and then uploads the wheels to the staging area. (This could even be done via an automated script, or even a fully automated build bot).
• once all the wheels have been uploaded, the main release manager double-checks things, and then hits the switch to make the release live
• the source and binary releases appear on pypi in a single atomic transaction

We've had problems in the past with numpy where we used a similar workflow but directly on the real pypi, so that there was a period between steps one and two where the source release was live and the binary release was not. Since pip prefers a 1.10.3 sdist to a 1.10.2 wheel, any users who try to install during this gap suddenly revert from the wheel experience to the far far inferior sdist experience. The solution of course is to coordinate everything offline and like email wheels back and forth to collect them all on a single person's machine before doing the upload in one go. But this is difficult and unpleasant.

[dstufft]

You don't need to collect onto a single users machine. You just need to upload all of the wheels first but that's still obviously not as nice of an experience. I'm not opposed to this feature though. It's just not a priority at the moment.

Sent from my iPhone

On Jan 7, 2016, at 4:32 PM, Nathaniel J. Smith notifications@github.com wrote:

Another advantage of this feature would be that it would provide a way for distributed teams to coordinate releases. E.g. one could use a workflow like:

main release manager uploads (hopefully) final source release to one of these staging areas
the volunteer who provides osx builds fetches the source from this staging area, builds wheels, and then uploads the wheels to the staging area. (This could even be done via an automated script, or even a fully automated build bot).
once all the wheels have been uploaded, the main release manager double-checks things, and then hits the switch to make the release live
the source and binary releases appear on pypi in a single atomic transaction
We've had problems in the past with numpy where we used a similar workflow but directly on the real pypi, so that there was a period between steps one and two where the source release was live and the binary release was not. Since pip prefers a 1.10.3 sdist to a 1.10.2 wheel, any users who try to install during this gap suddenly revert from the wheel experience to the far far inferior sdist experience. The solution of course is to coordinate everything offline and like email wheels back and forth to collect them all on a single person's machine before doing the upload in one go. But this is difficult and unpleasant.

—
Reply to this email directly or view it on GitHub.

[sigmavirus24]

It also means there's an extra step on top of twine upload

[njsmith]

We hesitated on just uploading wheels before the sdist first because we had no idea what pip would do if it saw that a 1.10.3 release was available, but only in formats that can't be used by the current machine. I'm actually not sure whether in this case I'd prefer pip to error out or to silently fall back to an earlier version, though obviously the fallback is better for handling this particular situation. And you also still have the problem of how to distribute the sdist to the build volunteers if not via pypi.

But yeah, it's manageable. The main advantage of this approach is the testing use case described in the OP; I just wanted to make a note that it is also have other advantages. (And of course these are complementary -- you can use the same staging area to first assemble the wheels and then to test the whole assemblage.) Numpy's had to skip release numbers twice in the last few months due to issues interacting with pypi. I'm not saying this is pypi's fault -- I think one was user error and one was a network error -- but rather just, there's a lot of advantages to reducing the cost of errors.

[dstufft]

That's easy enough to manage. Just have it default to immediate publish with a flag to enable the two step upload.

Sent from my iPhone

On Jan 7, 2016, at 4:38 PM, Ian Cordasco notifications@github.com wrote:

It also means there's an extra step on top of twine upload

—
Reply to this email directly or view it on GitHub.

[dstufft]

It'll install the older sdist.

Sent from my iPhone

On Jan 7, 2016, at 5:45 PM, Nathaniel J. Smith notifications@github.com wrote:

We hesitated on just uploading wheels before the sdist first because we had no idea what pip would do if it saw that a 1.10.3 release was available, but only in formats that can't be used by the current machine.

[ncoghlan]

@dstufft, from a twine UX perspective, I take it you mean something like twine upload --no-publish <artifacts>? I think that would work well, especially if it wrote out the private URL so you could subsequently do twine publish <unpublished release URL>

[dstufft]

@ncoghlan Yea something like that. We'll want to consider what it means if/when we move twine into pip too. But either way I think it's reasonably doable in a way that doesn't change the default behavior but makes it easy to opt in.