Encoded data is encoded once more

[forman]

With input.decode_cf set to false (the default), nc2zarr loads encoded data into memory. The first time a variable is written, its encoded data is correctly written as-is even if it contains the scaling_factor or add_offset attributes. But when nc2zarr (actually the xarray.to_zarr() method) attempts to append a new time step to existing data, it uses the existing variable encoding information to encode the already encoded data before writing it. This leads to corrupt datasets if scaling_factor or add_offset attributes are used.

The actual issue has its root cause in xarray.to_zarr() method that behaves inconsistently. A new encode_cf keyword argument would also help.

[forman]

xarray as of version 0.18.2 always CF-encodes variable data according to encoding information of existing variables before it appends new data. If the data to be appended is already encoded (because by default we read it with dataset = xr.open_dataset(..., decode_cf=False)) then this leads to entirely corrupt data. A possible hack is to redundantly dataset = decode_cf(dataset) before appending so encoding again is finally a no-op. This of course also costs extra CPU.

A better solution is to fix xarray's inconsitent dataset.to_zarr() behaviours when either creating variables or appending data to existing variables. I'll post an xarray issue. For example, when I uncomment line 469 in xarray/backends/zarr.py, the this fixes our issue too:

https://github.com/pydata/xarray/blob/1b4412eeb7011f53932779e1d7c3534163aedd63/xarray/backends/zarr.py#L460-L471