Implement shift for CFTimeIndex

[aidanheerdegen]

Code Sample

import numpy as np
import xarray as xr
import pandas as pd

from cftime import num2date, DatetimeNoLeap

times = num2date(np.arange(730), calendar='noleap', units='days since 0001-01-01')
da = xr.DataArray(np.arange(730), coords=[times], dims=['time'])

Problem description

I am trying to shift a time index as I need to align datasets to a common start point.

Directly incrementing one of the CFTimeIndex values works:

>>> da.time.get_index('time')[0] + pd.Timedelta('365 days')
cftime.DatetimeNoLeap(2, 1, 1, 0, 0, 0, 0, -1, 1)

Trying to use shift does not:

>>> da.time.get_index('time').shift(1,'Y')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/g/data3/hh5/public/apps/miniconda3/envs/analysis3-18.04/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2629, in shift
    type(self).__name__)
NotImplementedError: Not supported for type CFTimeIndex

If I want to shift a time index is the only way currently is to loop over all the individual elements of the index and add a time offset to each.

Expected Output

I would expect to have CFTimeIndex shifted by the desired time delta.

Output of xr.show_versions()

# Paste the output here xr.show_versions() here

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.17.1.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C
LOCALE: None.None

xarray: 0.10.7
pandas: 0.23.1
numpy: 1.14.5
scipy: 1.1.0
netCDF4: 1.3.1
h5netcdf: 0.5.1
h5py: 2.8.0
Nio: None
zarr: 2.2.0
bottleneck: 1.2.1
cyordereddict: None
dask: 0.17.5
distributed: 1.21.8
matplotlib: 1.5.3
cartopy: 0.16.0
seaborn: 0.8.1
setuptools: 39.2.0
pip: 9.0.3
conda: None
pytest: 3.6.1
IPython: 6.4.0
sphinx: None

[aidanheerdegen]

Submitted issue brought up in this thread

#2191 (comment)

[spencerkclark]

@aidanheerdegen thanks, I agree an implementation of shift would be nice. In the meantime, if the offset can be written as a fixed amount of time (like 365 days as opposed to something like one month), a slightly simpler way than looping over the elements of the index is to add a datetime.timedelta object to it, e.g.:

In [1]: import numpy as np

In [2]: import xarray as xr

In [3]: from cftime import num2date

In [4]: from datetime import timedelta

In [5]: xr.set_options(enable_cftimeindex=True)
Out[5]: <xarray.core.options.set_options at 0x110885a10>

In [6]: times = num2date(np.arange(730), calendar='noleap', units='days since 0001-01-01')

In [7]: da = xr.DataArray(np.arange(730), coords=[times], dims=['time'])

In [8]: da.time.get_index('time')
Out[8]:
CFTimeIndex([0001-01-01 00:00:00, 0001-01-02 00:00:00, 0001-01-03 00:00:00,
             0001-01-04 00:00:00, 0001-01-05 00:00:00, 0001-01-06 00:00:00,
             0001-01-07 00:00:00, 0001-01-08 00:00:00, 0001-01-09 00:00:00,
             0001-01-10 00:00:00,
             ...
             0002-12-22 00:00:00, 0002-12-23 00:00:00, 0002-12-24 00:00:00,
             0002-12-25 00:00:00, 0002-12-26 00:00:00, 0002-12-27 00:00:00,
             0002-12-28 00:00:00, 0002-12-29 00:00:00, 0002-12-30 00:00:00,
             0002-12-31 00:00:00],
            dtype='object', name=u'time', length=730)

In [9]: da.time.get_index('time') + timedelta(days=365)
Out[9]:
Index([0002-01-01 00:00:00, 0002-01-02 00:00:00, 0002-01-03 00:00:00,
       0002-01-04 00:00:00, 0002-01-05 00:00:00, 0002-01-06 00:00:00,
       0002-01-07 00:00:00, 0002-01-08 00:00:00, 0002-01-09 00:00:00,
       0002-01-10 00:00:00,
       ...
       0003-12-22 00:00:00, 0003-12-23 00:00:00, 0003-12-24 00:00:00,
       0003-12-25 00:00:00, 0003-12-26 00:00:00, 0003-12-27 00:00:00,
       0003-12-28 00:00:00, 0003-12-29 00:00:00, 0003-12-30 00:00:00,
       0003-12-31 00:00:00],
      dtype='object', length=730)

Note, however, that currently the index type is not preserved under this sort of arithmetic (which we might consider a separate issue). This can be alleviated by casting the result as a CFTimeIndex:

In [10]: xr.CFTimeIndex(da.time.get_index('time') + timedelta(days=365))
Out[10]:
CFTimeIndex([0002-01-01 00:00:00, 0002-01-02 00:00:00, 0002-01-03 00:00:00,
             0002-01-04 00:00:00, 0002-01-05 00:00:00, 0002-01-06 00:00:00,
             0002-01-07 00:00:00, 0002-01-08 00:00:00, 0002-01-09 00:00:00,
             0002-01-10 00:00:00,
             ...
             0003-12-22 00:00:00, 0003-12-23 00:00:00, 0003-12-24 00:00:00,
             0003-12-25 00:00:00, 0003-12-26 00:00:00, 0003-12-27 00:00:00,
             0003-12-28 00:00:00, 0003-12-29 00:00:00, 0003-12-30 00:00:00,
             0003-12-31 00:00:00],
            dtype='object', length=730)

[aidanheerdegen]

Great! Thanks @spencerkclark

I agree this is an excellent work around.

One of the most frustrating aspect of using an otherwise brilliant tool like xarray (and other python packages) is knowing what NOT to try and do. It is otherwise almost magical in the things it does well, so there is something of an expectation that it can and should do everything. If this work-around was part of the official documentation (until shift is implemented) that would be very useful.

I am often in the position of recommending software solutions to scientists who need to analyse their data, and the date issue with xarray (which CFTime has mostly alleviated) always made me hesitant to recommend it whole-heartedly. So thanks for your part in adding this to xarray.