Error with to_netcdf() when DataArray attribute is a list of strings

[smithara]

MCVE Code Sample

(without netCDF4 installed)

from xarray import DataArray
da = DataArray(attrs={"A": ["a"]})
da.to_netcdf("file.nc")

gives KeyError from scipy.io.netcdf:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-110-d933f8a2641b> in <module>
      1 da = DataArray(attrs={"A": ["a"]})
      2 print(da)
----> 3 da.to_netcdf("file.nc")
      4 # da

~/miniconda3_envs/py37/lib/python3.7/site-packages/xarray/core/dataarray.py in to_netcdf(self, *args, **kwargs)
   2332             dataset = self.to_dataset()
   2333 
-> 2334         return dataset.to_netcdf(*args, **kwargs)
   2335 
   2336     def to_dict(self, data: bool = True) -> dict:

~/miniconda3_envs/py37/lib/python3.7/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1545             unlimited_dims=unlimited_dims,
   1546             compute=compute,
-> 1547             invalid_netcdf=invalid_netcdf,
   1548         )
   1549 

~/miniconda3_envs/py37/lib/python3.7/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1086     finally:
   1087         if not multifile and compute:
-> 1088             store.close()
   1089 
   1090     if not compute:

~/miniconda3_envs/py37/lib/python3.7/site-packages/xarray/backends/scipy_.py in close(self)
    221 
    222     def close(self):
--> 223         self._manager.close()

~/miniconda3_envs/py37/lib/python3.7/site-packages/xarray/backends/file_manager.py in close(self, needs_lock)
    219             file = self._cache.pop(self._key, default)
    220             if file is not None:
--> 221                 file.close()
    222 
    223     def __del__(self):

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in close(self)
    297         if hasattr(self, 'fp') and not self.fp.closed:
    298             try:
--> 299                 self.flush()
    300             finally:
    301                 self.variables = OrderedDict()

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in flush(self)
    407         """
    408         if hasattr(self, 'mode') and self.mode in 'wa':
--> 409             self._write()
    410     sync = flush
    411 

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in _write(self)
    419         self._write_dim_array()
    420         self._write_gatt_array()
--> 421         self._write_var_array()
    422 
    423     def _write_numrecs(self):

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in _write_var_array(self)
    467             # Set the metadata for all variables.
    468             for name in variables:
--> 469                 self._write_var_metadata(name)
    470             # Now that we have the metadata, we know the vsize of
    471             # each record variable, so we can calculate recsize.

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in _write_var_metadata(self, name)
    488             self._pack_int(dimid)
    489 
--> 490         self._write_att_array(var._attributes)
    491 
    492         nc_type = REVERSE[var.typecode(), var.itemsize()]

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in _write_att_array(self, attributes)
    448             for name, values in attributes.items():
    449                 self._pack_string(name)
--> 450                 self._write_att_values(values)
    451         else:
    452             self.fp.write(ABSENT)

~/miniconda3_envs/py37/lib/python3.7/site-packages/scipy/io/netcdf.py in _write_att_values(self, values)
    560     def _write_att_values(self, values):
    561         if hasattr(values, 'dtype'):
--> 562             nc_type = REVERSE[values.dtype.char, values.dtype.itemsize]
    563         else:
    564             types = [(t, NC_INT) for t in integer_types]

KeyError: ('U', 4)

Expected Output

Saves the netcdf file, or gives an error about an unsupported situation?

Problem Description

I don't know if this is a bug in scipy.io.netcdf or if this type of attribute is not meant to be supported by netcdf, or it's a bug in xarray to_netcdf.

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.7.2 (default, Dec 29 2018, 06:19:36) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1062.1.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.4 libnetcdf: None

xarray: 0.14.1
pandas: 0.25.3
numpy: 1.15.4
scipy: 1.1.0
netCDF4: None
pydap: None
h5netcdf: 0.6.2
h5py: 2.9.0
Nio: None
zarr: 2.2.0
cftime: 1.0.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 0.20.0
distributed: 1.23.3
matplotlib: 3.0.2
cartopy: 0.17.0
seaborn: None
numbagg: None
setuptools: 40.4.3
pip: 18.1
conda: 4.5.11
pytest: 4.1.1
IPython: 7.0.1
sphinx: 1.8.3

[keewis]

I think this is a limitation of engine="scipy"? Using engine="netcdf4" or engine="h5netcdf" don't fail...

[smithara]

(sorry, forgot to update this)

Yeah that's right. I can't remember where I read it but it's related to a limitation in either the netcdf3 format or scipy's implementation. The quick fix is to pip install netCDF4 then the above code works as expected (as xarray prioritises engine="netcdf4" if it is available) .

[keewis]

good to know you found this out yourself. I'm closing this issue, then.