NotImplementedError: Opaque datatype class not supported after last commit

[bushaev]

Hey!
I have this piece of code

def extract_offset(path, dset_name="_index"):
    # find index dataset and extract offset
    f = None
    try:
        f = pyfive.File(path)
        dset = f.get(dset_name)
        if dset is None:
            return None
        else:
            return dset._dataobjects.offset

    finally:
        if f is not None:
            f.close()

That used to work until the last merged PR, now it gives me this:

Traceback (most recent call last):
File "/workspace/cli/jobs/P3_split_audio.py", line 175, in <module>
run_task_cli(
File "/workspace/cli/common/job_utils.py", line 157, in run_task_cli
with artifact_manager_class(
File "/workspace/cli/common/artifacts/artifact_manager.py", line 190, in __exit__
artifact.save()
File "/workspace/cli/common/artifacts/split_audio.py", line 12, in save
make_index(self.h5_path)
File "/workspace/cli/common/artifacts/hdf5_indexer.py", line 91, in make_index
update_offset(path, dset_name)
File "/workspace/cli/common/artifacts/hdf5_indexer.py", line 26, in update_offset
offset = extract_offset(path, dset_name=dset_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/cli/common/artifacts/hdf5_indexer.py", line 14, in extract_offset
dset = f.get(dset_name)
^^^^^^^^^^^^^^^^
File "<frozen _collections_abc>", line 774, in get
File "/customer/.local/lib/python3.11/site-packages/pyfive/high_level.py", line 94, in __getitem__
return Dataset(obj_name, DatasetID(dataobjs), self)
^^^^^^^^^^^^^^^^^^^
File "/customer/.local/lib/python3.11/site-packages/pyfive/h5d.py", line 85, in __init__
if isinstance(dataobject.dtype,tuple):
^^^^^^^^^^^^^^^^
File "/customer/.local/lib/python3.11/site-packages/pyfive/dataobjects.py", line 355, in dtype
return DatatypeMessage(self.msg_data, msg_offset).dtype
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/customer/.local/lib/python3.11/site-packages/pyfive/datatype_msg.py", line 16, in __init__
self.dtype = self.determine_dtype()
^^^^^^^^^^^^^^^^^^^^^^
File "/customer/.local/lib/python3.11/site-packages/pyfive/datatype_msg.py", line 36, in determine_dtype
raise NotImplementedError("Opaque datatype class not supported.")
NotImplementedError: Opaque datatype class not supported.

Could you advise on how to fix the error ?

[valeriupredoi]

hi @bushaev cheers for dropping by and opening the issue! Could I get a Minimum Reproducible Example (MRE) code (just like the above one, but one that also includes the call to load the data), and a sample of your data so I can try replicate the issue, please? Cheers 🍺

[bnlawrence]

I think if it has stopped working, then whatever is causing this error now was ignored before. If you can provide a piece of data with that in it, we can work out how to carry on ignoring it, if that's what the problem was. I don't believe any our our changes should have stopped some data that was previously visible from being visible, so it'd be good to have some data to test that assertion. One thing you could do is put a small dataset into a test that fails, and give us a pull request for that? (Failing that, just drop us a bit of data, and if possible, the code that created it.)

[kmuehlbauer]

I'm having issues with that too. I'm currently in the process to make h5netcdf's testsuite accept pyfive as given in @davidhassell's PR h5netcdf/h5netcdf#273.

Unfortunately pyfive raises NotImplementedError also for ENUMERATED. In my case the issue is, that although I can work around the direct access of such variable with a try/except. This doesn't work for h5netcdf's netcdf4-based dimension lookup, which works via REFERENCE type. When dereferencing these pyfive is running through visititems, actually instanciating everything

pyfive/pyfive/high_level.py

Lines 127 to 140 in 7772b9b

	def visititems(self, func):
	"""
	Recursively visit all objects in this group and subgroups.
	func should be a callable with the signature:
	func(name, object) -> None or return value
	Returning None continues iteration, return anything else stops and
	return that value from the visit method.
	"""
	root_name_length = len(self.name)
	if not self.name.endswith('/'):

and throwing the NotImplementedError.

I've tried to wrap my head around and have come up with some krude hack for now, which I'm opening a PR for, so that you can have a look.

My suggestion would be instead of throwing a NotImplementedError, gracefully return something meaningful. Maybe just a placeholder variable?

[kmuehlbauer]

Please have a look at #83 for details.

[bnlawrence]

I think possibly the solution here is to ask "why is visititems" visiting everything for a reference look up? This is likely the source of performance problems we have noted with dimensions as well. (I guess we know the answer, but maybe there is a way to avoid this and/or cache it so it is more efficient and avoids raising errors unless folks attempt to access something which is not supported.)

[kmuehlbauer]

The reference is looked up by its address:

    def _get_object_by_address(self, obj_addr):
        """ Return the object pointed to by a given address. """
        if self._dataobjects.offset == obj_addr:
            return self
        return self.visititems(
            lambda x, y: y if y._dataobjects.offset == obj_addr else None)

So if the object is not the actual object, visititems is called to check the address until found. If in that process a not implemented object is encountered, it raises.

[bnlawrence]

Ok, this needs to be handled more gracefully. Opaque data types are not something we really want to do much with. It looks like h5py can create some opaque data type for us to generate some proper tests around.

[bnlawrence]

The reference is looked up by its address:

def _get_object_by_address(self, obj_addr):
    """ Return the object pointed to by a given address. """
    if self._dataobjects.offset == obj_addr:
        return self
    return self.visititems(
        lambda x, y: y if y._dataobjects.offset == obj_addr else None)

So if the object is not the actual object, visititems is called to check the address until found. If in that process a not implemented object is encountered, it raises.

Yes, that looks like we should try and avoid doing that in any case where we don't expect to be able to read the object itself, so you're hack is definitely in the right direction. It might be we want it to be even more general though ... See also @bmaranville's comment here.