Resample is ~100x slower than Pandas resample; Speed is related to resample period (unlike Pandas)

[mankoff]

What happened:

I have a 10 minute frequency time series. When I resample to hourly it is slow. When I resample to daily it is fast. If I drop to Pandas and resample the speeds are ~100x faster than xarray, and also the same time regardless of the resample period. I've posted this to SO: https://stackoverflow.com/questions/64282393/

What you expected to happen:

I expect xarray to be within an order of magnitude speed of Pandas, not > 2 orders of magnitude slower.

Minimal Complete Verifiable Example:

import numpy as np
import xarray as xr
import pandas as pd
import time

size = 10000
times = pd.date_range('2000-01-01', periods=size, freq="10Min")
da = xr.DataArray(data = np.random.random(size), dims = ['time'], coords = {'time': times}, name='foo')

start = time.time()
da_ = da.resample({'time':"1H"}).mean()
print("1H", 'xr', str(time.time() - start))

start = time.time()
da_ = da.to_dataframe().resample("1H").mean()
print("1H", 'pd', str(time.time() - start), "\n")


start = time.time()
da_ = da.resample({'time':"1D"}).mean()
print("1D", 'xr', str(time.time() - start))

start = time.time()
da_ = da.to_dataframe().resample("1D").mean()
print("1D", 'pd', str(time.time() - start))

Output/timings

: 1H xr 0.1761918067932129
: 1H pd 0.0021948814392089844 
: 
: 1D xr 0.00958395004272461
: 1D pd 0.001646280288696289

Anything else we need to know?:

Environment:

Output of xr.show_versions()

xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.5 | packaged by conda-forge | (default, Aug 21 2020, 18:21:27)
[GCC 7.5.0]
python-bits: 64
OS: Linux
OS-release: 5.4.0-48-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.7.4

xarray: 0.16.0
pandas: 1.1.1
numpy: 1.19.1
scipy: 1.5.2
netCDF4: 1.5.4
pydap: None
h5netcdf: 0.8.1
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.2.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.5
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.3.1
cartopy: None
seaborn: None
numbagg: None
pint: 0.15
setuptools: 49.6.0.post20200814
pip: 20.2.2
conda: None
pytest: None
IPython: 7.17.0
sphinx: None

[dcherian]

@mankoff (hi!) This is interesting. If I comment out .mean() I get

1H xr 0.003030538558959961
1H pd 0.0014064311981201172 

1D xr 0.0026717185974121094
1D pd 0.0013244152069091797

i.e. we are 2x slower just on factorizing.

[mankoff]

"performance" is a good tag. My actual use case is a dataset with 500,000 timestamps and 15 variables (10 minute weather station for a decade).

In this case, pandas takes 0.03 seconds, and xarray takes 200 seconds. 4 orders of magnitude. Should I change the title to reflect the larger difference in performance? Here is that MWE:

import numpy as np
import xarray as xr
import pandas as pd
import time

size = 500000
times = pd.date_range('2000-01-01', periods=size, freq="10Min")
ds = xr.Dataset({
    'foo': xr.DataArray(
        data   = np.random.random(size),
        dims   = ['time'],
        coords = {'time': times}
    )})
for v in 'abcdefghijelm':
    ds[v] = (('time'), np.random.random(size))

start = time.time()
ds_r = ds.resample({'time':"1H"}).mean()
print('xr', str(time.time() - start))

start = time.time()
ds_r = ds.to_dataframe().resample("1H").mean()
print('pd', str(time.time() - start))

Result:

xr 202.2967929840088
pd 0.03381085395812988

The strange thing here is if I drop the .mean()'s, most of the time I see what you see.

: xr 0.03333306312561035
: pd 0.020237445831298828

But every 4th or 5th time that I run this, I get this:

: xr 0.8518760204315186
: pd 0.02686452865600586

This is repeatable. I've Run this code 100s of times now, and every 4th or 5th run it takes 10x. Nothing else is going on on my computer.

[mankoff]

The every 4th or 5th lag is not in the creation, it's in the resample:

#+BEGIN_SRC jupyter-python :kernel ds :session bugreport
for i in np.arange(25):
    start = time.time()
    ds_r = ds.resample({'time':"1H"})
    print('xr', str(time.time() - start))

#+END_SRC

#+RESULTS:
#+begin_example
xr 0.04479050636291504
xr 0.047682762145996094
xr 0.8904871940612793
xr 0.05605506896972656
xr 0.0452876091003418
xr 0.0467374324798584
xr 0.8709239959716797
xr 0.05595755577087402
xr 0.046492576599121094
xr 0.04648017883300781
xr 0.045223236083984375
xr 0.8187246322631836
xr 0.05060911178588867
xr 0.04763054847717285
xr 0.8156075477600098
xr 0.055490970611572266
xr 0.047312259674072266
xr 0.04651069641113281
xr 0.8001837730407715
xr 0.05546212196350098
xr 0.04549074172973633
xr 0.04680013656616211
xr 0.04383039474487305
xr 0.7662224769592285
xr 0.04914355278015137
#+end_example

[shoyer]

resample uses the same machinery in xarray as other grouped aggregations.

Right now, grouped aggregations are very slow when there are many groups (like in resample) because we use a Python loop over groups.

Probably the most obvious way to speed this up would be to wrap the "numpy-groupies" package in xarray: #4473

[shoyer]

I might add that this is somewhat I've wanted to speed-up in xarray since the very early days. But until I noticed the numpy-groupies package, it seemed like a pretty challenging task.

[mankoff]

Thanks for the clarification that this is a real issue not due to just my coding, and the suggestion to solve this elsewhere. For now I just use the fast Pandas version with this code:

df_h = ds.to_dataframe().resample("1H").mean()  # what we want (quickly), but in Pandas form
vals = [xr.DataArray(data=df_h[c], dims=['time'], coords={'time':df_h.index}, attrs=ds[c].attrs) for c in df_h.columns]
ds_h = xr.Dataset(dict(zip(df_h.columns,vals)), attrs=ds.attrs)

[mankoff]

The linked issues refer to groupby not resample so this could stay open or be closed as a duplicate - I leave it to you to decide. Thank you for the assistance.