Arbitrary chunking of uncompressed files (e.g. netCDF3)

[TomNicholas]

@rabernat made the interesting point to me that uncompressed files (e.g. netCDF3 files) have no specific chunking, as you can start reading bytes from any point in the file immediately, with no minimum unit of decompression to do first.

I'm not totally sure what this implies for VirtualiZarr generating references from netCDF3 files, as it's still meaningful to talk about one chunk per file concatenated together in a manifest.

Perhaps this is something that should ultimately be taken account of in zarr readers: that reading bytes from an uncompressed array does not require loading an entire chunk into memory first.

[TomNicholas]

It seems great minds think alike https://medium.com/pangeo/using-kerchunk-with-uncompressed-netcdf-64-bit-offset-files-cloud-optimized-access-to-hycom-ocean-9008ba6d0d67

That post uses kerchunk.utils.subchunk (which I did not know existed), but I wonder if a neater API to use for a similar feature in virtualizarr could just be to add a chunks argument to open_virtual_dataset, that can only be used for uncompressed files?

[TomNicholas]

In fact one could imagine a .rechunk method on a ManifestArray, but it would raise unless the ManifestArray pointed to contiguous-on-disk uncompressed data.