Work around for CFGRIB memory leak?

[emfdavid]

Has anyone found a work around for the memory leak in CFGrib?

My best solution right now is running my scangrib calls in Dask and then get dask to kill the workers when they go over limit.

I am building a kerchunked version of HRRR output in GCP.
Anyone interested in access iff I can make it public?

[martindurant]

I was not previously aware of the memory leak. grib_scan is due for an overhaul either way. One possibility if to use eccodes directory rather than cfgrib, as gribscan does. Likewise, cogrib shows that it's possible to use cfgrib without local files, and presumably not write .idx files, if indeed that is the underlying issue.

Either/both of those approaches could be incorporated to what happens here, or you can use them as is to produce kerchunk-compatible output.

[emfdavid]

Okay - if you figure out a path forward between those two and possibly fixing cfgrib - I might be able to help implement it.
For now I will set low memory termination thresholds on my dask workers and add a periodic callback to delete the junk in /tmp. I filled up a 100GB VM in a few hours this afternoon.

[martindurant]

Do you see the same leak when accessing the data after successful kerchunking? The codec GRIBCodec uses cfgrib on the file parts, so maybe it suffers the same problem.

In the long term, it is feasible to try to actually decode the contents of grib messages directly without cfgrib/eccodes, but the format is complex and that would take some effort.

[martindurant]

Can you please check with #198 ?