Inline loaded variables into kerchunk references

[TomNicholas]

Would close case (1) of #62.

After #69 was merged, the next step (this PR) is to write loaded numpy arrays into the kerchunk references "inlined".

[TomNicholas]

The important part of this PR now works, but there's some annoying inconsistency in how kerchunk writes out NaN vs "NaN" vs null which is causing the test to fail 😠

[TomNicholas]

For reference the reason this test fails is that kerchunk.hdf.SingleHdf5ToZarr produces a references dict with these entries:

{
    'air/.zarray': '{"chunks":[2920,25,53],"compressor":null,"dtype":"<i2","fill_value":null,"filters":null,"order":"C","shape":[2920,25,53],"zarr_format":2}',
    ...
    'lat/.zarray': '{"chunks":[25],"compressor":null,"dtype":"<f4","fill_value":"NaN","filters":null,"order":"C","shape":[25],"zarr_format":2}',
...
}

Notice that for the variable air the default zarr fill_value of np.NaN is encoded as null, whereas for the variable lat it is encoded as "NaN". I see no possible reason for this discrepancy, or what the pattern is, so I can't really make my code consistent with inconsistent behaviour 🤷‍♂️ The fsspec mapper apparently accepts either.

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 90.49%. Comparing base (f226093) to head (cd6ca47).
Report is 23 commits behind head on main.

❗ Current head cd6ca47 differs from pull request most recent head 0baee1a. Consider uploading reports for the commit 0baee1a to get more accurate results

Files	Patch %	Lines
virtualizarr/kerchunk.py	85.71%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #73      +/-   ##
==========================================
+ Coverage   90.18%   90.49%   +0.31%     
==========================================
  Files          14       15       +1     
  Lines         998     1021      +23     
==========================================
+ Hits          900      924      +24     
+ Misses         98       97       -1     

Flag	Coverage Δ
unittests	90.49% <93.75%> (+0.31%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TomNicholas]

Whilst this PR now passes for "inlining" the lat and lon variables (yay), it will fail if you attempt to load either the air variable or the time variable.

The air variable case is some json encoding error, which should be fixable with the right set of arguments to json.dumps.

The time variable case is I think much harder - the encoded bytes look different to those that come out of kerchunk, presumably because kerchunk is doing some kind of CF-related encoding first. But looking inside kerchunk.hdf5.SingleHdfToZarr I am none the wiser as to what it's doing that I would need to replicate.

[TomNicholas]

This would be cool because we could apply the encoding to every block in parallel (using map_blocks with #39), and that would allow the user to choose their chunking in the kerchunk output just by rechunking this variable using xarray before serializing it.

[TomNicholas]

Also doing this for only some of the chunks would effectively implement case (2) of #62.

[TomNicholas]

The time variable case is I think much harder - the encoded bytes look different to those that come out of kerchunk, presumably because kerchunk is doing some kind of CF-related encoding first. But looking inside kerchunk.hdf5.SingleHdfToZarr I am none the wiser as to what it's doing that I would need to replicate.

So if I simply pass decode_times=False to xr.open_dataset internally then this test passes.
I think this implies that kerchunk is not actually doing any decoding of times at all?

[dcherian]

https://fsspec.github.io/kerchunk/reference.html#kerchunk.combine.MultiZarrToZarr

”cf:{var}”, interpret the value of var using cftime, returning a datetime. 
These will be automatically re-encoded with cftime, unless you specify an “M8[*]” 
dtype for the coordinate, in which case a conversion will be attempted.

[TomNicholas]

I have no idea what this is:

>       expected = SingleHdf5ToZarr(netcdf4_file, spec=1, inline_threshold=500).translate()

virtualizarr/tests/test_integration.py:8: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/kerchunk/hdf.py:93: in __init__
    self._h5f = h5py.File(self.input_file, mode="r")
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/h5py/_hl/files.py:562: in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/h5py/_hl/files.py:235: in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
    ???
h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   OSError: Unable to synchronously open file (file signature not found)

Given that the tests all pass for me locally

[TomNicholas]

So right now I can successfully inline (i.e. match kerchunk) the lat and lon variables in the xarray tutorial dataset (which have no encoding), but cannot serialize the air variable (which has scale_factor and offset encoding), nor the time variable (which has cf calendar and units encoding).

So I've parametrized the test to show what works and xfailed the ones that don't. I'm also now raising NotImplementedErrors when encoding is encountered.

[TomNicholas]

@norlandrhagen do you have any idea what this error is? Or even how I might try to reproduce it locally?

OSError: Unable to synchronously open file (file signature not found)

[TomNicholas]

@martindurant I'm seeing an error in a call to SingleHdf5ToZarr (which I'm calling for comparison purposes) - but it's only occurring in the CI. All my google results for this error say that it means the file is corrupted, but I don't understand why it would be corrupted when written out from xarray (and openable locally). Have you seen this before?

EDIT: Nevermind @norlandrhagen came to my rescue! ❤️