Issue loading file due to buffer too small

[valeriupredoi]

Here is the MRE:

import pyfive


def _load_nc_file(ncvar):
    """
    Get the netcdf file and its b-tree.

    Fixture to test loading an issue file.
    """
    issue_file = "da193a_25_6hr_t_pt_cordex__198807-198807.nc"
    nc = pyfive.File(issue_file)
    ds = nc[ncvar]

    return ds


def test_buffer_issue():
    print("File issue: da193a_25_6hr_t_pt_cordex__198807-198807.nc")
    print("Variable m01s30i111")
    _load_nc_file('m01s30i111')

test_buffer_issue()

this instantly returns the barf:

$ python test_buffer_local.py
File issue: da193a_25_6hr_t_pt_cordex__198807-198807.nc
Variable m01s30i111
Traceback (most recent call last):
  File "/home/valeriu/test_buffer_local.py", line 22, in <module>
    test_buffer_issue()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/valeriu/test_buffer_local.py", line 20, in test_buffer_issue
    _load_nc_file('m01s30i111')
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/valeriu/test_buffer_local.py", line 12, in _load_nc_file
    ds = nc[ncvar]
         ~~^^^^^^^
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/high_level.py", line 94, in __getitem__
    return Dataset(obj_name, DatasetID(dataobjs), self)
                             ~~~~~~~~~^^^^^^^^^^
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/h5d.py", line 93, in __init__
    self._meta = DatasetMeta(dataobject)
                 ~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/h5d.py", line 593, in __init__
    self.attributes = dataobject.get_attributes()
                      ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/dataobjects.py", line 154, in get_attributes
    more_attrs = self._get_attributes_from_attr_info(attrs, attr_info)
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/dataobjects.py", line 180, in _get_attributes_from_attr_info
    name, value = self._parse_attribute_msg(data,0)
                  ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/dataobjects.py", line 237, in _parse_attribute_msg
    value = self._attr_value(dtype, buffer, items, offset)
  File "/home/valeriu/miniconda3/envs/pyactive4/lib/python3.13/site-packages/pyfive/dataobjects.py", line 273, in _attr_value
    value = np.frombuffer(buf, dtype=dtype, count=count, offset=offset)
ValueError: buffer is smaller than requested size

This is run locally on POSIX, but in #99 the same behaviour occurs for the same file but on an S3 FS, accessed via s3fs. This has to be something specific to the file, I will investigate more, and will post updates below, in the comments.

[kmuehlbauer]

@valeriupredoi Can you make an h5dump -A from that file?

[valeriupredoi]

@kmuehlbauer on it now, from netCDF4.Dataset I gathered: it's a NETCDF4, with source: HadGEM3-GC31-MM (2016), CF standards CF 1.10 - so looks kosher to me, also @bnlawrence was asking

[valeriupredoi]

@kmuehlbauer here is the h5dump dump

h5dump_buffer_issue_file.txt

[valeriupredoi]

by comparison here is the h5dump of a comparable file, but that doesn't barf the buffer barf

h5dump_ok2_file.txt

[kmuehlbauer]

Can't spot anything obviously wrong here (on the phone). Need to check more closely. Can the offending dataset or attribute be identified?

[valeriupredoi]

Can't spot anything obviously wrong here (on the phone). Need to check more closely. Can the offending dataset or attribute be identified?

sames, man - am looking through vs the dump from the other file that works and I really can't see anything obvious. The offending vars are any of the vars in the file...

[valeriupredoi]

Can't spot anything obviously wrong here (on the phone). Need to check more closely. Can the offending dataset or attribute be identified?

sames, man - am looking through vs the dump from the other file that works and I really can't see anything obvious. The offending vars are any of the vars in the file... - no I am lying, some variables work, so far I found these to work:

• b_t_um_atmos_DALLTH
• b_bnds_t_um_atmos_DALLTH
• lon
• um_atmos_DALLTH

[valeriupredoi]

so all other DATASETS work fine apart from DATASET "m01s00i010_9" and DATASET "m01s30i111"

[kmuehlbauer]

Both datasets have this in common (DATASPACE NULL).

ATTRIBUTE "coordinates" {
   DATATYPE  H5T_STRING {
      STRSIZE 1;
      STRPAD H5T_STR_NULLTERM;
      CSET H5T_CSET_ASCII;
      CTYPE H5T_C_S1;
   }
   DATASPACE  NULL
   DATA {
   }
}

[valeriupredoi]

yes that looks very odd, there is another file I am looking at containing tas as variable, that file works very well, and it has coordinates as attribute to the variable, but like this:

      ATTRIBUTE "coordinates" {
         DATATYPE  H5T_STRING {
            STRSIZE 6;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "height"
         }
      }