oldest-supported-numpy Erroneously Required in Apple Silicon Python 3.10 Wheel

[william-silversmith]

Hi zfpy team,

Thanks again for uploading my wheels! I was attempting to use them and ran into an issue that I found was my own doing. During the wheel compilation, I had added install_requires=['oldest-supported-numpy', 'cython'] to setup.py while I was experimenting, but forgot to remove it. This is causing a version conflict with some other software I am using that requires a newer version of numpy. I'm happy to produce a set of fixed wheels if that would be okay. I've developed a superior build process since I last tried this.

# build_macos_universal2_libzfp.sh
#!/usr/bin/sh

ARMDIR=build_macos/arm64
X86DIR=build_macos/x86_64

mkdir -p $ARMDIR
mkdir -p $X86DIR

cmake -B $ARMDIR -S . -DCMAKE_OSX_ARCHITECTURES=arm64 -DBUILD_TESTING=NO -DBUILD_SHARED_LIBS=OFF
cmake -B $X86DIR -S . -DCMAKE_OSX_ARCHITECTURES=x86_64 -DBUILD_TESTING=NO -DBUILD_SHARED_LIBS=OFF

DIR=$(pwd)
cd $DIR/$ARMDIR && make
cd $DIR/$X86DIR && make

lipo -create -output $DIR/libzfp.a $DIR/$ARMDIR/lib/libzfp.a $DIR/$X86DIR/lib/libzfp.a

# setup-macos.py
from setuptools import setup, Extension

class NumpyImport:
  def __repr__(self):
    import numpy as np

    return np.get_include()

  __fspath__ = __repr__

setup(
    name="zfpy",
    version="1.0.0",
    author="Peter Lindstrom",
    author_email="zfp@llnl.gov",
    url="https://zfp.llnl.gov",
    description="zfp compression in Python",
    long_description="zfp is a compressed format for representing multidimensional floating-point and integer arrays. zfp provides compressed-array classes that support high throughput read and write random access to individual array elements. zfp also supports serial and parallel compression of whole arrays using both lossless and lossy compression with error tolerances. zfp is primarily written in C and C++ but also includes Python and Fortran bindings.",
    ext_modules=[
        Extension("zfpy", ["python/zfpy.pyx"],
            include_dirs=["include", NumpyImport()],
            library_dirs=['/Users/wms/code/zfp/'],
            libraries=["zfp"]
        )
    ]
)

# tox.ini
[tox]
envlist = py38,py39,py310,py311

[testenv]
platform = darwin
deps = 
	oldest-supported-numpy
	cython

commands = 
	python setup.py bdist_wheel

To actually build the wheels:

sh ./build_macos_universal2_libzfp.sh
tox

The key difference here from my previous build process is that I have finally figured out how to get tox to run on setup.py files that require numpy. This ensures a clean build.

[lindstro]

@da-wad Can you please comment on this?

[da-wad]

Hi, I'm happy to create (another) post release with updated wheels from you, but want to be confident that we don't create other problems. Ideally we want wheels which work with every supported version of numpy, so does building with a newer numpy mean it is still possible to use the wheels with an older numpy? Or is there a way to insist on the "oldest-supported-numpy" being a minimum version?

[william-silversmith]

Hi,

Perhaps this is not the most rigorous way to do it, but the way I've done it in the past was to insist on using oldest-supported-numpy during the building of wheels (i.e. in the github action script) and allow the numpy requirement for installation to be free-floating. Everything was fine the way things were being done, I just did something dumb while building the 3.10 wheel manually and made it so that pip would insist on using the exact oldest-supported-numpy version during use, not just during building.

You can also just steal the current parameters from oldest-supported-numpy and put them in the requirements as numpy >= VERSION

[tasansal]

This is causing me a big problem with dependency resolution with Poetry. It causes a lot of unnecessary checks when generating lock files. It causes an explosion of dependency resolution checks and takes an hour to generate a lock file (instead of 5 seconds). Suprisingly, sometimes it even conflicts with numpy requirements from matplotlib and the lock file generation fails.

I also would support having a numpy >= VERSION instead of oldest-supported-numpy.

@william-silversmith anything I can help with to push this through?

[fcollman]

@da-wad @william-silversmith What can we do to resolve this concern? Installing things with zfpy is presently pretty problematic with python 3.10 right now due to this issue.

[william-silversmith]

I can at least try building some universal wheels on my end and post them here.

[william-silversmith]

Hi all,

Here are some updated wheels with the version bumped to 1.0.1 so that if you re-upload them (along with a new build) people will automatically have them installed as the newest version. Let me know if you'd like to try something different re: versioning.

zfpy-1.01-universal2-wheels.zip

[tasansal]

@lindstro @da-wad How can we resolve this? 1.0.0 dependency resolution doesn't even work with Python 3.11.

Like @fcollman mentioned it does cause many issues with installation.

I believe the universal wheels by @william-silversmith will solve the issue given Zfp releases a 1.0.1 version.

Thanks a lot.

[lindstro]

I don't know all the issues involved, though I'd be willing to cut a 1.0.1 release if that fixes the problem. It's however unclear to me what changes to zfp would be needed. Is it just a matter of bumping the version, or do we also need to make changes to the zfpy build scripts? Are there any interactions with #210 that we need to consider?

[tasansal]

I don't know all the issues involved, though I'd be willing to cut a 1.0.1 release if that fixes the problem. It's however unclear to me what changes to zfp would be needed. Is it just a matter of bumping the version, or do we also need to make changes to the zfpy build scripts? Are there any interactions with #210 that we need to consider?

Just to push this forward, let me add some comments :) @william-silversmith please chime in if I'm wrong.

1. Shouldn't affect any ZFP business logic. This is purely for Python wrapper on Mac wheels systems.
2. I think the build scripts may need to be updated. @william-silversmith has the scripts he used to build the ARM64 Mac wheels in the issue description. I'm not sure how these play with existing build scripts.
3. Shouldn't affect Numpy 2.0 #210; the erroneous oldest-supported-numpy addition only makes things break, and it has no upside in this issue. Zfpy can make a specific numpy version minimum/maximum without the need for oldest-supported-numpy.

[william-silversmith]

Yes, essentially it just needs a new release with the new binaries uploaded. The previous version of them suffered from a mistake I made while building manually. The wheels are built with oldest-supported-numpy (and so are compatible with future numpy versions) but no longer require using the oldest supported numpy version during installation.

[tasansal]

@lindstro @william-silversmith, can we please fix this one way or another? I am unsure what exactly needs to be done if you all tell me I can make a PR! Or is it just a matter of making new wheels?

[william-silversmith]

It's really just a matter of bumping the version and releasing new wheels, incl. the ones I posted above.