Support Enumerated Data Types

[bnlawrence]

The enumeration is a particular case of a compound data type, so particular that it has it's own data type message. Apparently enumerations are used in some NetCDF4 contexts, so we should probably support them.

[bnlawrence]

Notes: It seems that h5py doesn't know a dataset (aka variable) is an enumeration without checking that it is one, it only knows that it is a uint8, and the interpretation that it has metadata (values) associated with the integer keys stored in the actual integer variable is only known by doing a check_enum_dtype on the variable dtype itself, which will return a dictionary of the enumeration, or None. Some thoughts on what to do next are in this comment.

[bnlawrence]

Reading an enumeration is rather straightforward I think (draft code - 85e2441 - in at in branch enum_and_friends. (Might be doing this fast enough to deal with #83 as well.)

[bmaranville]

I don't know if you can see the comments I made on 85e2441 , but it seems like the "base dtype" that appears to be 4 bytes in the specification document seems to be a full, embedded datatype message for the base type instead, and so you could just call "self.determine_dtype()" again once you've read the ENUM datatype_msg, to get the base dtype, and this will automatically advance the offset to the start of the names property.

Then someday if we add support for the bit offset and bit precision fields in the fixed type, they'll automatically work in ENUM too!

[bnlawrence]

Thanks for looking at this. As it happens I stumbled across the same information in parallel with your writing these comments, and I've implemented something similar as you've seen. I see you've noted another problem which I'll need to think about. I'm currently trying to work out what's going on the attribute handling where I think h5py has got lazy. My test case seems to have (courtesy of using netcdf4-python to generate the data) a pointer of sorts to the enum in an attribute ... I'm likely to have to give up on this shortly, but I'll commit something as I head off.

[bnlawrence]

Ok, sadly I really have to pause work on this until the first week of September, but I'm quite close. The issue at hand is a decision around how much to implement of the h5py h5py DataType (and possibly h5t.TypeNumID) we want to implement. That's necessary so instead of returning None for a DataType we return something. I suspect there will be a couple of things we need to do to the actual dataobject dtype for the dataset which is of type enum, but i also suspect that will be trivial.

(There's a subsidiary issue in that I've implemented something different than h5py in the handling of attributes which are intended to point to the enum dictionary, it looks like h5py just fluffs it, whereas I've returned the actual enum dictionary - the key words in that sentence are "looks like" which should not be construed as "actually").

[bnlawrence]

Overnight I realised that for the subsidiary issue above, we should really return a Datatype instance, not the dictionary (the dictionary can be obtained from the Datatype both via the check_enum_dtype function, and via direct inspection (without deeply inspecting the h5py code, it looks like the check_enum_dtype function simply iterates over the members of the dictionary by using the methods of the Datatype instances's id attribute (which is a TypeNumID instance).

[bnlawrence]

The latest commit shows that I had misunderstood a few things when I wrote those last two comments. In particular, it turns out that the enumeration dictionary is available as metadata on the numpy dtype itself! We will need to make this clear in the documentation.

[bnlawrence]

I've pretty much finished this, and am ready to do a pull request. But it'd be useful @kmuehlbauer if you could have a look at the documentation and make sure I've interpreted enum usage correctly. It looks ok, but I've reverse engineered this all from the h5py docs and inspecting the h5py objects after reading enums, so could have taken a wrong turn or two.

[kmuehlbauer]

@bnlawrence Thanks, the docs look nice and I think catch all the bit's and pieces.

Just to get some idea of how this looks like, this is an h5dump of an enum_type and an enum_var (with transient type):

   DATATYPE "enum_t" H5T_ENUM {
      H5T_STD_U8LE;
      "missing"          255;
      "one"              1;
      "three"            3;
      "two"              2;
   };
   DATASET "enum_var" {
      DATATYPE  H5T_ENUM {
         H5T_STD_U8LE;
         "missing"          255;
         "one"              1;
         "three"            3;
         "two"              2;
      }
      DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
      DATA {
      (0): one, two, three, missing
      }
      ATTRIBUTE "_FillValue" {
         DATATYPE  H5T_ENUM {
            H5T_STD_U8LE;
            "missing"          255;
            "one"              1;
            "three"            3;
            "two"              2;
         }
         DATASPACE  SCALAR
         DATA {
         (0): missing
         }
      }
   }