Skip to content

Non-kerchunk backend for GRIB files. #312

Open
@sharkinsspatial

Description

@sharkinsspatial

I did a bit of investigation into wrapping the existing kerchunk grib reader to create a GRIB backend but discovered that kerchunk forces inlining of derived coordinates that are not stored at the message level. Virtualizarr does not support reading inlined refs. This might be a good reason to kick off work on a Virtualizarr specific backend without a direct kerchunk dependency (which was an eventual goal).

I personally have limited experience working with GRIB internals so it would be valuable to get input here from someone with deeper experience like @mpiannucci. A few questions

  1. Should we consider https://github.com/mpiannucci/gribberish for ChunkManifest generation (will it see continued development/maintenance)?
  2. I'm leaning towards a model where we use gribberish as an optional dependency in Virtualizarr and place the backend code in this project rather than generating ChunkManifests via gribberish as is currently done for kerchunk refs via scan_gribberish but I don't have strong opinions on this.
  3. Can we assume coordinate alignment across all messages in a GRIB file and use open_virtual_datatree or should our we also include an open_virtual_groups method for problematic datasets? @mpiannucci we'd also probably need some recommendations for from you about documentation we can include on the types of concat and grouping operations users would perform on the dict returned by open_virtual_groups.

ref #11 ref #238

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions