NumPy 2.0 support

[jakirkham]

NumPy 2.0 is coming out soon ( numpy/numpy#24300 ). NumPy 2.0.0rc1 packages for conda & wheels came out 3 weeks back ( numpy/numpy#24300 (comment) )

To prepare for NumPy 2.0, it might be worthwhile to start testing against NumPy 2 in CI

NumPy has put out a migration guide. More details are in the release notes

If Numcodecs make use of NumPy's C API (and produces wheels that use it), having a release of Numcodecs with wheels built against NumPy 2.0.0rc1 would be helpful to ensure NumPy 1 & 2 compatible wheels (as wheels built against NumPy 1 won't be compatible with NumPy 2). More details in this NumPy 2 ABI doc

Also as NumPy is tracking ecosystem support for NumPy 2.0, it would be helpful to share Numcodecs current support status in issue (with any plans): numpy/numpy#26191

[jakirkham]

As Numcodecs has relied primarily on the Python Buffer Protocol and not really used NumPy's C API, this shouldn't involve rebuilding Numcodecs. Have just searched through the code to reconfirm this

The thing to check for would be if Numcodecs is affected by any NumPy Python API breaks. This could be done by adding it to CI here

[jakirkham]

FWIW attempted running the test suite locally with NumPy 2.0.0rc1 from conda-forge and it appeared to pass. So doesn't look like any changes are needed

Still would be good to add to CI

[jakirkham]

We do have an optional dependency on zfpy

numcodecs/pyproject.toml

Lines 66 to 68 in 4abe4be

	zfpy = [
	"zfpy>=1.0.0",
	]

zfpy still needs to do some work to migrate: LLNL/zfp#210

However this just means users won't be able to install zfpy until that is resolved. No impact on installing Numcodecs itself

[dstansby]

It might be worth doing a release with the numpy<2 pin first (which is now in main) while this gets sorted out?

[joshmoore]

👍 Certainly what is happening in several other of my repos.

Edit: see https://github.com/zarr-developers/numcodecs/pull/528/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711

[rgommers]

Didn't @jakirkham confirm above already that everything works as expected for numcodecs? If so, no need to add a pin now. Based on that testing, we marked numcodecs as compatible in numpy/numpy#26191 a long time ago.

The linked PR (gh-535) only shows test failures because of zfpy, which @jakirkham had also commented on already.

[dstansby]

👍 without zfpy looks like it tests work fine with numpy 2.0.

[jakirkham]

Yep all we needed was CI. Thanks for adding that! 🙏

NumPy 2 support in zfpy is a separate issue. Presumably this would be solved with upstream PR ( LLNL/zfp#231 ) and a new tag. In any event it is not blocking for numcodecs