Numpy string types in compound type

[jacklovell]

I have the following Numpy dtype for an array:

dtype([('Object_type', 'S30'), ('ID', 'S30'), ('Version', '<i4'), 
('basis_1', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')]),
('basis_2', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')]), 
('centre_point', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')]), 
('width', '<f8'), ('height', '<f8'),
('slit_id', 'S30'), ('slit_no', '<i4')])

When I try to create a NetCDF compound type from this dtype, I get the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
netCDF4/_netCDF4.pyx in netCDF4._netCDF4._def_compound (netCDF4/_netCDF4.c:51191)()

KeyError: 'S30'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-247-c1ab3d42293b> in <module>()
     20     nc_coord = bolo_group.createCompoundType(cartesian_coord, "COORDINATE")
---> 21     nc_detector = bolo_group.createCompoundType(aperture_dtype_full, "DETECTOR")
     22     foil_no = bolo_group.createDimension("foil_no", foils.size)

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.createCompoundType (netCDF4/_netCDF4.c:16268)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.CompoundType.__init__ (netCDF4/_netCDF4.c:49971)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._def_compound (netCDF4/_netCDF4.c:51245)()

ValueError: Unsupported compound type element

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-247-c1ab3d42293b> in <module>()
     27     nc_slits = sxd_group.createVariable("slits", nc_detector, "slit_no")
     28     nc_slits[:] = slits_full
---> 29     nc_slits.units = "mm"

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__exit__ (netCDF4/_netCDF4.c:13090)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.close (netCDF4/_netCDF4.c:15045)()

RuntimeError: NetCDF: HDF error

I understand that it is not possible to create compound types which include variable-length arrays, but the Numpy '<S30' type should always be 30 bytes, not a variable length. This is actually why I'm using S30 and not U30, since the latter may have multi-byte characters.

Is it possible to add support for the S30 dtype in netcdf4-python? Otherwise, is there another way that I can include fixed length string types in a NetCDF compound type?

[jswhit]

Problem is there is no fixed length string datatype in netcdf. The only workaround I know of is to create a character array ('S1') with a length of 30, and then use the stringtoarr and chartostring utilities to convert back and forth from strings to arrays of characters. There's an example of this for compound types at https://github.com/Unidata/netcdf4-python/blob/master/examples/tutorial.py.

In your case, you would use

dtype([('Object_type', 'S1',30), ('ID', 'S1',30), ('Version', '<i4'), 
('basis_1', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')]),
('basis_2', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')]), 
('centre_point', [('x', '<f8'), ('y', '<f8'), ('z', '<f8')]), 
('width', '<f8'), ('height', '<f8'),
('slit_id', 'S1',30), ('slit_no', '<i4')])

I've asked for fixed length strings to be added to netcdf-c in the past, but the idea never gained any traction (Unidata/netcdf-c#132). If you really want this, please comment on that ticket.

[jswhit]

BTW - it is technically possible to support vlen strings inside compound data types. I just have devoted the time to implement it (yet).

[jacklovell]

Thanks for this. Using the strtoarr and chartostring utilities works well, although it does add some boilerplate to the code.

It would be nice if there was at least an option in netcdf4-python to perform this step internally. Xarray does this (http://xarray.pydata.org/en/stable/io.html#string-encoding) for its netcdf IO, although that doesn't seem to work for compound data types yet (pydata/xarray#1977). Enabling this at the netcdf4-python level would make it easier for other packages to support what would be a very useful feature.

[jswhit]

We already to this for character arrays if the _Encoding attribute is set (that is what xarray is using under the hood), so it may not be too hard to support for compound types. I'll look into it.

[shoyer]

One option is to use .view() to convert the data from repeated single characters to strings. Consider:

In [9]: data = [('a', 'bb'), ('ccc', 'dddd')]

In [10]: arr1 = np.array(data, dtype=[('f0', 'S1', 3), ('f1', 'S1', 4)])

In [11]: arr1
Out[11]:
array([([b'a', b'a', b'a'], [b'b', b'b', b'b', b'b']),
       ([b'c', b'c', b'c'], [b'd', b'd', b'd', b'd'])],
      dtype=[('f0', 'S1', (3,)), ('f1', 'S1', (4,))])

In [12]: arr2 = np.array(data, dtype='S3,S4')

In [13]: arr2
Out[13]:
array([(b'a', b'bb'), (b'ccc', b'dddd')],
      dtype=[('f0', 'S3'), ('f1', 'S4')])

In [14]: arr3 = arr1.view(arr2.dtype)

In [15]: arr3
Out[15]:
array([(b'aaa', b'bbbb'), (b'ccc', b'dddd')],
      dtype=[('f0', 'S3'), ('f1', 'S4')])

[shoyer]

OK, numpy is doing something horrible with arr1 -- those element should not be repeated like that.

But hopefully you get the idea that view() can be used to convert between data types.

[jswhit]

Yes, I think views are the simplest and safest way to do this. You just have to create two numpy datatypes, one with the string components represented as character arrays, and one with numpy string arrays. Just use the first (dtype1) to create the netcdf variable, and the second (dtype2) to create your data, and write the data to the variable using v[:] = data[:].view(dtype1). To read the data use data[:] = v[:].view(dtype2).

[jswhit]

An example

from netCDF4 import Dataset
import numpy as np
f = Dataset('compound_example.nc','w')
dtype1 = np.dtype([('observation', 'f4'),
                    ('station_name','S1',80)])
dtype2 = np.dtype([('observation', 'f4'),
                    ('station_name','S80')])
station_data_t = f.createCompoundType(dtype1,'station_data')
f.createDimension('station',None)
statdat = f.createVariable('station_obs', station_data_t, ('station',))
data = np.empty(2,dtype2)
data['observation'][:] = (123.,3.14)
data['station_name'][:] = ('Boulder','New York')
statdat[:] = data.view(dtype1)
print statdat[:]
print
print statdat[:].view(dtype2)
f.close()

[(123.  , ['B', 'o', 'u', 'l', 'd', 'e', 'r', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ''])
 (  3.14, ['N', 'e', 'w', ' ', 'Y', 'o', 'r', 'k', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ''])]

[(123.  , 'Boulder') (  3.14, 'New York')]

ncdump compound_example.nc

netcdf compound_example {
types:
  compound station_data {
    float observation ;
    char station_name(80) ;
  }; // station_data
dimensions:
	station = UNLIMITED ; // (2 currently)
variables:
	station_data station_obs(station) ;
data:

 station_obs = {123, {"Boulder"}}, {3.14, {"New York"}} ;

[jswhit]

Pull request #778 enables this automatically, so now this works

from netCDF4 import Dataset
import numpy as np
f = Dataset('compound_example.nc','w')
dtype = np.dtype([('observation', 'f4'),
                  ('station_name','S1',80)])
station_data_t = f.createCompoundType(dtype,'station_data')
f.createDimension('station',None)
statdat = f.createVariable('station_obs', station_data_t, ('station',))
statdat._Encoding = 'ascii'
data = np.empty(2,station_data_t.dtype_view)
print data.dtype
data['observation'][:] = (123.,3.14)
data['station_name'][:] = ('Boulder','New York')
statdat[:] = data
print statdat.dtype
print statdat[:].dtype
print statdat[:]
f.close()

{'names':['observation','station_name'], 'formats':['<f4','S80'], 'offsets':[0,4], 'itemsize':84, 'aligned':True}
{'names':['observation','station_name'], 'formats':['<f4',('S1', (80,))], 'offsets':[0,4], 'itemsize':84, 'aligned':True}
{'names':['observation','station_name'], 'formats':['<f4','S80'], 'offsets':[0,4], 'itemsize':84, 'aligned':True}
[(123.  , 'Boulder') (  3.14, 'New York')]
~

[jswhit]

I'm a little nervous about adding this extra magic. The pros are:

1. it's only done if the _Encoding variable attribute is set, so the behavior is similar to what happens with character array variables except that a view is returned instead of a copy.
2. simplifies user code a bit, since the user almost always wants numpy strings and not character arrays. If the user really wants the character arrays, they can just not set _Encoding.

Cons:

1. may be confusing to get numpy data back that is not the same type as the netcdf variable.
2. may break existing code (doubtful, since probably no one is setting _Encoding on compound types).

[jswhit]

@jacklovell and @shoyer, I really would like your feedback on this proposed change.

[shoyer]

Remind me -- does the _Encoding attribute get set automatically?

I do think this is probably a win for usability. Most users want NumPy strings, not arrays of characters.

[jswhit]

Pull request was updated so that if you specify 'S#' in a structured dtype when creating a netcdf compound type it automatically gets converted to ('S1',#). So now in the above example

dtype = np.dtype([('observation', 'f4'),
                  ('station_name','S1',80)])
station_data_t = f.createCompoundType(dtype,'station_data')

can be changed to

dtype = np.dtype([('observation', 'f4'),
                 ('station_name','S80')])
station_data_t = f.createCompoundType(dtype,'station_data')

_Encoding does not get set automatically to preserve backward compatibility. To get the new behavior, you have to explicitly set it.

[jacklovell]

@jswhit This looks good. Under-the-hood conversion is better in my opinion, since it removes the need for boilerplate in users' codes (particularly having to create 2 similar-but-not-quite-identical dtypes). Being able to pass in a Numpy dtype with strings and read back a Numpy dtype with strings, without having to manually convert 'S#' to ('S1', #), does make this more user friendly, I think.