Decoding int32 arrays with CF-Conventions applied.

[dwest77a]

What happened?

Attempting to decode a masked int32 encoded array with CF-Conventions applied, specifically in the CFMaskCoder.decode function.

What did you expect to happen?

Expected behaviour is that masked int32 variables are decoded correctly, given the correct data type and fill value to replace with masks.

Issue is specific to CFMaskCoder.decode line 370, where for a masked array of int32 values, the following check gives an incorrect outcome:

data.dtype.kind in "iu" should return False for signed integers, but instead returns True since "i" in "iu" also returns True. The fix should then be to replace this check with data.dtype.kind in ["iu"] to apply the dtype int64 to unsigned integer values, not signed 32bit integers.

Minimal Complete Verifiable Example

#Example dataset can be downloaded from https://data.ceda.ac.uk//neodc/esacci/sst/data/CDR_v2/ATSR/L3U/v2.1/AATSR/2002/07/24/20020724123321-ESACCI-L3U_GHRSST-SSTskin-AATSR-CDR2.1-v02.0-fv01.0.nc


import xarray as xr

ds = xr.open_dataset('20020724123321-ESACCI-L3U_GHRSST-SSTskin-AATSR-CDR2.1-v02.0-fv01.0.nc')

ds.sst_dtime.isel(time=0, lat=slice(0,5), lon=slice(0,5)).compute()

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

This is the expected output, provided the CF-Conventions extend to int32 masked arrays. If this is not the case, the data can be decoded with decode_cf=False correctly, however this should not be needed if the CF conventions should apply.

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0] python-bits: 64 OS: Linux OS-release: 5.14.0-503.38.1.el9_5.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: ('en_GB', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.3-development

xarray: 2024.6.0
pandas: 2.2.3
numpy: 1.26.4
scipy: 1.15.2
netCDF4: 1.6.5
pydap: None
h5netcdf: None
h5py: 3.13.0
zarr: 2.18.7
cftime: 1.6.4.post1
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.7.0
distributed: 2024.7.0
matplotlib: 3.9.2
cartopy: None
seaborn: None
numbagg: None
fsspec: 2024.9.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 75.9.1
pip: 24.0
conda: None
pytest: 8.3.5
mypy: None
IPython: 9.1.0
sphinx: 7.4.7

[welcome]

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

[keewis]

data.dtype.kind in "iu" should return False for signed integers

dtype.kind will return i for signed integers of any size (and "iu" is used as a short-hand for {"i", "u"}), so the code itself correctly represents the intention. With numpy>=2 it would probably be best written as np.isdtype(x, "integer").

The only question is whether that intention itself is wrong, which I'd have to look into a bit further to tell (or ask @kmuehlbauer or @spencerkclark).

For reference, the only instance of this at

xarray/xarray/coding/variables.py

Line 413 in 99a1ad2

) and data.dtype.kind in "iu":

which has to do with datetime decoding, but changed a bit since v2024.06.0

[dcherian]

Can you give us a reproducible example showing us the output, and the expected output please?

[kmuehlbauer]

The mentioned code path normally only gets called, if we have a datetime/timedelta with units-attribute (eg. "days since ...."/"days").

This decodes CF times into datetime64/timedelta64.

@dwest77a if you do not want to decode times, please set decode_times=False in the call of open_dataset.

[kmuehlbauer]

@dwest77a Extending from my previous answer the issue is with automatic timedelta decoding:

In #10101 (released in 2025.04.0) an informative FutureWarning has been added:

FutureWarning: In a future version, xarray will not decode timedelta values based on the presence of a timedelta-like units attribute by default. Instead it will rely on the presence of a timedelta64 dtype attribute, which is now xarray's default way of encoding timedelta64 values. To continue decoding timedeltas based on the presence of a timedelta-like units attribute, users will need to explicitly opt-in by passing True or CFTimedeltaCoder(decode_via_units=True) to decode_timedelta. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.

For your case just add decode_timedelta=False:

import xarray as xr

ds = xr.open_dataset('20020724123321-ESACCI-L3U_GHRSST-SSTskin-AATSR-CDR2.1-v02.0-fv01.0.nc', decode_timedelta=False)

ds.sst_dtime.isel(time=0, lat=slice(0,5), lon=slice(0,5)).compute()

The second issue is xarray's inherent representation of masked arrays as float arrays with masked values represented as nan.

So your int32 data will be transformed to float. You can transform back to masked array like this:

ds.sst_dtime.isel(time=0, lat=slice(0,5), lon=slice(0,5)).to_masked_array()

But this does not preserve the correct _FillValue. For your usecase you might have to do the masking by yourself:

import numpy as np
import xarray as xr

ds = xr.open_dataset('20020724123321-ESACCI-L3U_GHRSST-SSTskin-AATSR-CDR2.1-v02.0-fv01.0.nc', decode_timedelta=False, mask_and_scale=False)

np.ma.masked_equal(ds.sst_dtime.isel(time=0, lat=slice(0,5), lon=slice(0,5)).compute(), np.iinfo("int32").min)

Integrating masked arrays directly into xarray has been discussed before. I remember @keewis mentioning https://github.com/mdhaber/marray/ in this context. He might shed more light on the efforts in this field.

[dwest77a]

@kmuehlbauer thanks very much for the detailed response. I can take these findings back to my organisation as answer to our question. The timedelta in this instance is not encoded with xarray in mind, hence these difficulties with normal use. Very good to have an explanation to go with this result. Thanks!

[keewis]

marray has been built to be compatible with the array API, so for data variables and non-indexed coordinates this should already work with xarray (this is untested, though, and we might need some convenience functions like ds.masked.where or ds.masked.drop_missing or similar).

For 1D data variables / coordinates I believe you can also use pandas extension arrays.