Possible `pyfive` bug/edge case

[mo-gill]

This issue is to cross reference this one from cmip7repack NCAS-CMS/cmip7_repack#21

I have been using check_cmip7_packing successfully for many files and it has been working great. However, on one particular file it fails with a CAN'T PARSE exit code of 5.

I have checked this file via a few different methods and it appears to be normal, and have successfully run the check on other similar files that have worked fine.

After having a dig the problem appears to occur in pyfive. Possibly some sort of edge case?

>>> import pyfive
>>> f = pyfive.File("filename")
>>> f.consolidated_metadata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<redacted>/lib/python3.10/site-packages/pyfive/high_level.py", line 272, in consolidated_metadata
    if isinstance(f[ds], Dataset):
  File "<redacted>/lib/python3.10/site-packages/pyfive/high_level.py", line 64, in __getitem__
    return self.__getitem_lazy_control(y, noindex=False)
  File "<redacted>/lib/python3.10/site-packages/pyfive/high_level.py", line 129, in __getitem_lazy_control
    return Dataset(obj_name, DatasetID(dataobjs, noindex=noindex), self)
  File "<redacted>/lib/python3.10/site-packages/pyfive/h5d.py", line 96, in __init__
    self._meta = DatasetMeta(dataobject)
  File "<redacted>/lib/python3.10/site-packages/pyfive/h5d.py", line 699, in __init__
    self.attributes = dataobject.get_attributes()
  File "<redacted>/lib/python3.10/site-packages/pyfive/dataobjects.py", line 155, in get_attributes
    more_attrs = self._get_attributes_from_attr_info(attrs, attr_info)
  File "<redacted>/lib/python3.10/site-packages/pyfive/dataobjects.py", line 181, in _get_attributes_from_attr_info
    name, value = self._parse_attribute_msg(data,0)
  File "<redacted>/lib/python3.10/site-packages/pyfive/dataobjects.py", line 197, in _parse_attribute_msg
    version = struct.unpack_from('<B', buffer, offset)[0]
struct.error: unpack_from requires a buffer of at least 1 bytes for unpacking 1 bytes at offset 0 (actual buffer size is 0)

We have hosted the file on JASMIN, if you would be able to investigate this it would be much appreciated:
~cdds/tmp/pyfive_error/

Thankyou!

[valeriupredoi]

many thanks @mo-gill 🍺 I'll have a looksee, I believe myself included, and @kmuehlbauer and @bnlawrence have seen issues with buffers that we fixed in the past

[valeriupredoi]

@mo-gill just before I proceed, could you pls tell me what version of pyfive you are using? I see all's done in a Python 3.10 environment, and that's a bit ye olde since we've now built cmip7_repack with Python>3.10

[valeriupredoi]

@mo-gill just before I proceed, could you pls tell me what version of pyfive you are using? I see all's done in a Python 3.10 environment, and that's a bit ye olde since we've now built cmip7_repack with Python>3.10

nevermind, I reproduced the issue with the now-migrated test case file in #160 - we'll be looking into it, I suspect a side-effect of #100

[kmuehlbauer]

@valeriupredoi @mo-gill

I've found the root case for this issue. The managed data of FractalHeap is not correctly fetched. Only the first row, 1024 bytes, is claimed.

Further, when accessing the attributes, the offsets are not decoded properly for attributes which are located in rows>0, which leads to issues when decoding.

I can have a first fix ready tomorrow.

[valeriupredoi]

@kmuehlbauer legend! My apologies for my tardiness, I'm shot down at home with the flu, so it's minimal work for me ATM. Kai, if you could pls pop the fix in #160 that'd be great so we can test with the exact file that @mo-gill pointed us to 🍻

[kmuehlbauer]

OK, I've pushed a fix to #160. @mo-gill If you are able, please test with your workflows.

Some more context:

The issue was, that three attributes have been written to a higher row in the FractalHeap. Until know, pyfive was not able to retrieve these parts of the FractalHeap. After enabling the correct retrieval, I had to implement a mapping from heapID based offsets to offsets into our flat managed buffer. Otherwise there were no means of accessing the attributes there. Fascinating, that we didn't catch this with the combined testsuites of pyfive, h5netcdf and xarray.

[kmuehlbauer]

My apologies for my tardiness, I'm shot down at home with the flu, so it's minimal work for me ATM.

No worries, @valeriupredoi! Get well soon!

[mo-gill]

OK, I've pushed a fix to #160. @mo-gill If you are able, please test with your workflows.

Some more context:

The issue was, that three attributes have been written to a higher row in the FractalHeap. Until know, pyfive was not able to retrieve these parts of the FractalHeap. After enabling the correct retrieval, I had to implement a mapping from heapID based offsets to offsets into our flat managed buffer. Otherwise there were no means of accessing the attributes there. Fascinating, that we didn't catch this with the combined testsuites of pyfive, h5netcdf and xarray.

I've just had a quick go using the fix and it seems to be working with both the problematic file (and the rest of our files in our testing suite) 🥳
Thankyou to you and everyone involved for this fix!