Skip to content

Install netcdf4 from conda-forge and pin netcdf4<1.7.3 in pyproject.toml#124

Merged
valeriupredoi merged 5 commits intomainfrom
pin_netcdf4
Oct 15, 2025
Merged

Install netcdf4 from conda-forge and pin netcdf4<1.7.3 in pyproject.toml#124
valeriupredoi merged 5 commits intomainfrom
pin_netcdf4

Conversation

@valeriupredoi
Copy link
Collaborator

@valeriupredoi valeriupredoi commented Oct 14, 2025

Description

Issues with the new released netCDF4==1.7.3 from PyPI
Closes #123

It is ONLY netcdf4==1.7.3 at fault since in an otherwise working env, simply installing 1.7.3 makes those tests fail:

(pyfive2) valeriu@valeriu-PORTEGE-Z30-C:~/pyfive$ pip install netcdf4==1.7.3
Collecting netcdf4==1.7.3
  Downloading netcdf4-1.7.3-cp311-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.9 kB)
Requirement already satisfied: cftime in /home/valeriu/miniconda3/envs/pyfive2/lib/python3.13/site-packages (from netcdf4==1.7.3) (1.6.5)
Requirement already satisfied: certifi in /home/valeriu/miniconda3/envs/pyfive2/lib/python3.13/site-packages (from netcdf4==1.7.3) (2025.8.3)
Requirement already satisfied: numpy in /home/valeriu/miniconda3/envs/pyfive2/lib/python3.13/site-packages (from netcdf4==1.7.3) (2.3.3)
Downloading netcdf4-1.7.3-cp311-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (9.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/9.5 MB 74.2 MB/s  0:00:00
Installing collected packages: netcdf4
  Attempting uninstall: netcdf4
    Found existing installation: netCDF4 1.7.2
    Uninstalling netCDF4-1.7.2:
      Successfully uninstalled netCDF4-1.7.2
Successfully installed netcdf4-1.7.3

Before you get started

Checklist

  • This pull request has a descriptive title and labels
  • This pull request has a minimal description (most was discussed in the issue, but a two-liner description is still desirable)
  • All tests pass

@codecov
Copy link

codecov bot commented Oct 14, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.05%. Comparing base (b673557) to head (dff9ccb).
⚠️ Report is 116 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #124   +/-   ##
=======================================
  Coverage   74.05%   74.05%           
=======================================
  Files          11       11           
  Lines        2598     2598           
  Branches      406      406           
=======================================
  Hits         1924     1924           
  Misses        565      565           
  Partials      109      109           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Collaborator

@kmuehlbauer kmuehlbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @valeriupredoi 🤞

@valeriupredoi
Copy link
Collaborator Author

cheers @kmuehlbauer - still something 🐟 y going on with Python 3.10 though, am looking at it now

@kmuehlbauer
Copy link
Collaborator

A weird issue, cftime 1.6.5 was also released recently as well as h5py 3.15. Hope you find a working environment. I've given up over at h5netcdf and feed almost everything from conda-forge.

@valeriupredoi
Copy link
Collaborator Author

the new wheel for 1.7.2 is causing it netcdf4-1.7.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl whereas the older wheel netCDF4-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl is working well 🤦‍♂️

@valeriupredoi
Copy link
Collaborator Author

A weird issue, cftime 1.6.5 was also released recently as well as h5py 3.15. Hope you find a working environment. I've given up over at h5netcdf and feed almost everything from conda-forge.

yes! That's the way forward, we even have pip install --no-deps in a few other packages; I'll have to do the same here, but we'd probably have to move to pixi to support actual optional dependencies

@valeriupredoi
Copy link
Collaborator Author

issues started coming in at netCDF4-python Unidata/netcdf4-python#1438

@valeriupredoi valeriupredoi changed the title pin netcdf4<1.7.3 Install netcdf4 from conda-forge and pin netcdf4<1.7.3 in pyproject.toml Oct 15, 2025
@valeriupredoi
Copy link
Collaborator Author

@kmuehlbauer here's our current solution: install netcdf4 from conda-forge, and pin it in pyproject.toml (so eager pip doesn't grab 1.7.3, which is not yet on conda-forge, the 1.7.3 PR is hitting a broken build) - do you partake? 🍺

@kmuehlbauer
Copy link
Collaborator

@valeriupredoi Maybe also add h5py to that list, so both libs link to the same hdf5.

@valeriupredoi
Copy link
Collaborator Author

@valeriupredoi Maybe also add h5py to that list, so both libs link to the same hdf5.

good call! Let me do it now

@valeriupredoi
Copy link
Collaborator Author

@valeriupredoi Maybe also add h5py to that list, so both libs link to the same hdf5.

good call! Let me do it now

done in 39b5de1

@kmuehlbauer
Copy link
Collaborator

Great sleuthing @valeriupredoi, this looks like it's working properly. One thing I do not understand from the install log: Why do we end up with ancient s3fs and other packages versions. Is this really intended behaviour?

@valeriupredoi
Copy link
Collaborator Author

Great sleuthing @valeriupredoi, this looks like it's working properly. One thing I do not understand from the install log: Why do we end up with ancient s3fs and other packages versions. Is this really intended behaviour?

lemme have a looksee, brb in 10min - good catch, I've not seen those yet

@valeriupredoi
Copy link
Collaborator Author

boto3 and botocore are now at the latest version - s3fs is indeed ancient though

Great sleuthing @valeriupredoi, this looks like it's working properly. One thing I do not understand from the install log: Why do we end up with ancient s3fs and other packages versions. Is this really intended behaviour?

lemme have a looksee, brb in 10min - good catch, I've not seen those yet

version mismatch with boto - we've been installing the superold s3fs for a long while now:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.40.52 requires botocore<1.41.0,>=1.40.52, but you have botocore 1.40.49 which is incompatible.
Successfully installed aiobotocore-2.25.0 aiohappyeyeballs-2.6.1 aiohttp-3.13.0 aioitertools-0.12.0 aiosignal-1.4.0 attrs-25.4.0 botocore-1.40.49 frozenlist-1.8.0 multidict-6.7.0 propcache-0.4.1 s3fs-2025.9.0 wrapt-1.17.3 yarl-1.22.0

it appears Amazon is not friends with fsspec much 😁

@valeriupredoi
Copy link
Collaborator Author

@kmuehlbauer fixed now, just don't ask me to run a pip check in the environment 😀

Copy link
Collaborator

@kmuehlbauer kmuehlbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm in favour to get this in, thanks @valeriupredoi 🥇

@valeriupredoi valeriupredoi merged commit 6573d26 into main Oct 15, 2025
6 checks passed
@valeriupredoi
Copy link
Collaborator Author

cheers @kmuehlbauer - @davidhassell it's always a LOT better to get all these netCDFy/h5py-y deps off conda-forge - one HDF5 to link against to, and that's it; as for the pin in pyproject.toml that I put here, that is most definitely temporary, and we can remove it once the 1.7.3 conda-forge package is out (in a few days, it seems); as for the pin on s3fs, it appears there's a long standing version mismatch boto/s3fs, I'd rather have us a modern s3fs and have to pin for it TBF 🍻

@valeriupredoi valeriupredoi deleted the pin_netcdf4 branch October 15, 2025 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI breaks with netCDF4 1.7.3 wheel

2 participants