Spec 0: How to handle versions with major API changes

[ngoldbaum]

Thinking about NumPy 2.0 coming out soon, I think it would be nice if the spec had some discussion about how downstream projects should handle major releases with significant API changes in packages they depend on.

[bsipocz]

The spec avoids referring to versions, major or minor, and is specific on the drop schedule being time-based. And says nothing about upper limits.

So not sure what we could/should suggest in regard of np 2.0

[joshmoore]

I'd be interested in thoughts/discussions of how we could include upcoming major changes either in SPEC-0 or in a separate SPEC. numpy/numpy#26191 is certainly a useful tool, but a clear message of the form "start testing alpha releases now" would be useful. Bonus points for having a central place (even an RSS feed 😄) for registering such broadcasts.

(Not knowing what I'm talking about, something between pip install and pip install --pre, e.g., pip install --next...)

[tupui]

The RSS (or else) feed is an interesting idea. One could subscribe to a list of projects and get all the relevant messages.

[bsipocz]

@joshmoore - a lot of projects are testing with nightly wheels, imo that is super helpful for avoiding surprises at dependency release time (and you could have that as a cron job, or allowed to fail to avoid failing unrelated PRs/ allow a bit more time for opening fixes)

[joshmoore]

@bsipocz: definitely! And I think they're great, but on the spectrum of possible upgrades to watch, that's even more churn than --pre, no?

   <-- Major -- Minor -- Patch -- Pre -- Nightly -->

Maybe this is almost more like distributions (e.g., eclipse, ubuntu) where projects opt in (for a particular build, etc.) to the "Next Big Thing" ™️ .

[rgommers]

I'm a little unclear why a third level would be needed. If you want the low-churn version, use --pre (alerts you at the beta stage). If you want the more active, higher-churn version, use a nightly.

For a major change like numpy 2.0, there'll be a beta version - 2.0.0b1 came out over 3 months ago. That was the signal here; anything before that is nightlies.

[joshmoore]

That may well be the case, and I'm very happy to hold my brainstorming for elsewhere/elsewhen. 😺 I think the issue I'm mulling over is: why didn't more testing take place? My guess would be that as a general rule fewer projects can take on the additional burden of constantly testing with --pre than we would like. I'm hypothesizing that if there was a less frequent, more critical coordination point that we could make a concerted push to test it widely. (Possibly with a more general rule about pinning <{next_major}...)

[bsipocz]

In practice, for the smaller libraries I maintain, there is hardly ever a lot to fix. So either --pre works or allowed to fail/cron nightlies.

OTOH, in my experience, if your project runs into a lot of problems then the only option is to pay attention to the nightlies and participate in the upstream dev cycle at least with feedback because you likely do something non-standard. (E.g astropy had a lot of issues with the new release, but it was not exactly unexpected, thus there was participation in the dev process, which imo was useful for upstream, too)

[rgommers]

What @bsipocz says above sounds about right to me. It doesn't get less burdensome than running a single job with --pre once in a while.

I think the issue I'm mulling over is: why didn't more testing take place?

Your impression may not be fully correct here. The amount of testing done for numpy 2.0 was enormous. If you're wondering why other packages didn't actually do a release to fix known problems - that's a different question I think. Doing releases can be a ton of work when you have to produce dozens of wheels, fix all your other problems in your main branch, etc. The majority of issues you're seeing is because packages didn't release in time.

[jarrodmillman]

The RSS (or else) feed is an interesting idea. One could subscribe to a list of projects and get all the relevant messages.

Maybe we should have a breaking tag on the blog for any upcoming breaking releases. Then folks could subscribe to:
https://blog.scientific-python.org/tags/breaking/atom.xml

[bsipocz]

The RSS sounds a possibly good idea, but is that really realistic that a project haven't heard about the upcoming breaking numpy 2.0 but will notice something on the RSS feed? It has been talked about for about a year now, and had an extremely long release cycle with betas expected to come out at the very end of last year/early this year which was then pushed forward due to the extended downstream testing.
(When a normal dev cycle is 6 months long having a pre-release out for 3 months is a really long time in my book)

So, my point is, that we really should advocate for best testing practices and a better adoption / normalization of SPEC4 .

[joshmoore]

rgommers commented 15 hours ago
Your impression may not be fully correct here. The amount of testing done for numpy 2.0 was enormous.

Huge apologies if I sound like I'm not appreciative of all the work and testing that did go on. But there are definitely libraries that did not do due diligence (raises hand in multiple cases) and so I'm trying to provide feedback on what may have helped catch these outliers. The best I have at the moment is a less stringent version of SPEC-4 so that it would receive wider adoption (or perhaps libraries for which SPEC-4 is all the more critical). But if I'm distracting from more key issues, I'm also happy to sit back and see how the rest of this lovely community moves things forward.

bsipocz commented 14 hours ago
is that really realistic that a project haven't heard about the upcoming breaking numpy 2.0 but will notice something on the RSS feed?

:) Probably not. And it might well be that nothing would have helped the stragglers. But I can tell you that on all of my projects I have definitely started pointing everyone to SPEC-0 in terms of when things are going to be dropped, and I had just noticed before posting #217 (comment) that numpy 2 wasn't mentioned there. Since I haven't yet found the time to implement SPEC-4, I was left in between. Certainly completely my own fault, but a situation that may well be indicative of others.

[rgommers]

No worries at all, no need to apologize. It'd be good to update SPEC 0 indeed to mention NumPy 2.0. It won't shift the regular time-based window at all though, so it's just an administrative update there.

Good point on SPEC 4: it could explain that if testing nightlies is too much work, then please still use --pre in a CI job with installs from PyPI rather than the nightlies bucket.

[joshmoore]

It'd be good to update SPEC 0 indeed to mention NumPy 2.0

👍

then please still use --pre in a CI job

❤️ and I will try to get there, but if you rely on even one library that misbehaves I can tell you this becomes problematic.

[ngoldbaum]

Maybe one thing we can do is write docs or develop some tooling to make it easier to run tests against nightlies with the following properties:

• The tests are allowed to fail
• New test failures against nightlies cause a bot to open a github issue about it which a human can triage and fix/ignore/open an upstream issue for.

I know a few projects do this and it seems to help.

[stefanv]

If I understand Josh's need, it is to have a central location where urgent updates are fed back to community developers. This could be on the blog, although I suspect a category, such as "developer notifications", on https://discuss.scientific-python.org would be easiest, since it has good custom subscription preferences.

Such a category would be low bandwidth, but can contain items like "be aware np 2 is breaking and is coming out soon", or "scipy sparse matrices now have an array-compatible interface" or "scikit-image is transitioning to a new package name, skimage2" (that hasn't happened, but you get the gist).

We can work references into whatever central announcement mechanism we choose into SPECs, to ensure wide awareness in the ecosystem.

[joshmoore]

stefanv commented 14 hours ago •
it is to have a central location where urgent updates are fed back to community developers

👍 if I can expand on that, I'd add coordination on the when and who has committed to update, which @rgommers did wonderfully in numpy/numpy#26191 but I missed. (https://python3statement.github.io/ also comes to mind.)

[jarrodmillman]

It'd be good to update SPEC 0 indeed to mention NumPy 2.0. It won't shift the regular time-based window at all though, so it's just an administrative update there.

#327