'numpy.datetime64' object has no attribute 'year' writing from grib2 source

[rsignell-usgs]

I have a consolidated JSON with a "best time series" dataset that reads from a collection of HRRR grib2 files on S3.

The data access and visualization of data in the notebook looks fine, but when I attempt to save a subset (to either zarr or netcdf) I get an error from xarray:

AttributeError: 'numpy.datetime64' object has no attribute 'year'

I'm raising this issue here because I run a nearly identical workflow with NetCDF files, where we also have datetime64 objects for time in xarray, and don't get this problem. Hence I'm thinking it must have something to do with the reading from grib2 files?

This notebook should be reproducible (and doesn't require AWS credentials):
https://nbviewer.org/gist/rsignell-usgs/7d0a76a5f9fbf5781e9e97b66bd72882

[martindurant]

I'll have a look when I have time, but my initial guess is that some attributes are left over from the original cftime encoding that is leaving xarray trying to make a conversion it shouldn't. Can you look at the attributes of the time variable? You might be able to solve by performing a trivial operation (+ timedelta(0)?) before save.

[martindurant]

The time column should definitely have been inlined in the references

Bytes | 1.38 kiB | 8 B

Maybe this is the problematic one, not valid_time, the coordinate, so you can remove it?

[rsignell-usgs]

I tried removing time and other potential problems by doing:

ds = ds.drop(['time', 'step', 'heightAboveGround']).rename({'valid_time':'time'})

but unfortunately still exact same error (you should be able to run this notebook locally without AWS credentials):

https://nbviewer.org/gist/5d10137059a1793f2606e39a5baf1ba2

[martindurant]

Can you also try multiplying by 1.0?

[martindurant]

(or subtracting timedelta(0) )

[rsignell-usgs]

@martindurant , I'm not sure I understand.
Multiply what by 1.0?
Before the json is created?

The datetime64 values are understood perfectly well in xarray -- the error only appears when persisted.
I don't understand why. Do you?
(should be easy for you to run yourself and check it out)

[martindurant]

I don't know why exactly, but I suspect that if you perform an arithmetic operation (like adding a timedelta zero), then any leftover encoding attributes might be usefully forgotten.

[rsignell-usgs]

@martindurant I'm still not sure what I'm supposed to multiply by 1.0 and at what point I'm supposed to do it.
Did you get a chance to try running the notebook that displays the error?

And here is the HRRR best time series notebook that created the json

[martindurant]

"forbidden" reading the JSON file. Do I need to run this in qhub?

[rsignell-usgs]

Grrr.... no, you don't need qhub. My bad. Just use this s_opts instead:

s_opts = {'anon':True, 'skip_instance_cache':True}

[martindurant]

ds2 = ds.assign(time=ds.time + pd.Timedelta(0))
ds2.isel(time=slice(-3,-1)).to_zarr('foo.zarr', 'w')

Allows you to write it, but xarray still wants to use some weird cftime encoding (even though the original data did not have that), causing overflow when you try to reload. Do you know how to not use cftime on write, just use actual datetime values? i.e., this is not a kerchunk problem, its an xarray/zarr encoding thing.

[rsignell-usgs]

@martindurant the CF-Conventions don't allow for datetime64 or even date strings: you need to supply float or int time values and a units attribute like "hours since 1990-01-01".

The xr.open_dataset() function has an argument we could set to use_cftime=False.

So you don't think this has anything to do with the grib2 codec or reading grib2 files?

[keewis]

this completes successfully:

subset = ds.assign_coords(time=ds.time.astype("datetime64[ns]")).isel(time=slice(-3,-1))
for var in subset.variables.values():
    var.encoding.pop("filters", None)
subset.to_zarr('foo.zarr', mode='w')

ds = xr.open_dataset("foo.zarr", engine="zarr").load()

cftime is only involved with dtypes that are not exactly datetime64[ns], so casting time avoids that. filters raised errors mentioning grib when I tried to load the resulting zarr.

[rsignell-usgs]

@keewis , whoa, thank you! I never would have figured this out! Is this something that users should handle using this approach or should we handle it in xarray?