Description
What happened?
Since numpy >= 2.0, xr.decode_cf() fails to replace FillValue with np.nan when:
- the data array dtype is
np.float32 - the
FillValueattribute is of typefloat.
The issue does not occur for data arrays with dtype np.float64.
This bug should not affect decoding of dataset opened with open_dataset because since numpy >= 2.0, numeric values in the attributes are now directly stored as np.int<...> or np.float32 types instead of the previously used int or float types.
To keep my code compatible with all numpy versions I currently use:
if version.parse(np.__version__) >= version.parse("2.0.0"):
vars_and_coords = list(ds.data_vars) + list(ds.coords)
for var in vars_and_coords:
if "_FillValue" in ds[var].attrs:
ds[var].attrs["_FillValue"] = ds[var].data.dtype.type(ds[var].attrs["_FillValue"])
What did you expect to happen?
xr.decode_cf() to replace fillvalues with np.nan also when the FillValue attribute is of type float.
Minimal Complete Verifiable Example
import numpy as np
import xarray as xr
fill_value = -99.9
data = np.ones((2,3)) * fill_value
data = data.astype(np.float32) # The error does not occur with float64 !
ds = xr.DataArray(data, dims=["x", "y"]).to_dataset(name="var")
ds["var"].attrs["_FillValue"] = fill_value # float
xr.decode_cf(ds)["var"].data # do not replace -99.9 values with np.nan
ds["var"].attrs["_FillValue"] = np.float32(fill_value) # np.float
xr.decode_cf(ds)["var"].data # do replace -99.9 values with np.nanMVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Environment
xarray: 2024.7.0
pandas: 2.2.2
numpy: 2.0.1
scipy: 1.14.0
netCDF4: 1.7.1
pydap: None
h5netcdf: None
h5py: 3.11.0
zarr: None
cftime: 1.6.4
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.8.1
distributed: 2024.8.1
matplotlib: 3.9.2
cartopy: 0.23.0
seaborn: None
numbagg: None
fsspec: 2024.6.1
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 72.1.0
pip: 24.2
conda: None
pytest: 8.3.2
mypy: None
IPython: 8.26.0
sphinx: 8.0.2