Xarray open_mfdataset is very slow

[etsmith14]

Xarray open_mfdataset is a fantastic tool, but I haven't been able to make it work efficiently. I am opening multiple NetCDFs each around 2GB for a total of 13GB. I have the data on an SSD, so it should open pretty quickly, however, it takes approx. 791 seconds to open the files. Any suggestions on how to speed this up? I have run several dozen tests with different options and different chunk sizes to try to zero in on the problem with no luck.

Below is the code I use to open the data:

UKESM_historical = xr.open_mfdataset(folder + r'\ukesm1-0-ll_r1i1p1f2_w5e5_historical_' + variable + '_global_daily_*.nc',
     concat_dim="time", data_vars='minimal', coords='minimal',
     compat='override', parallel=True, decode_cf = False, chunks={'time': 365}) # 791 seconds

[dcherian]

This should be quick if you only have 7 files (?). Can you try profiling with parallel=False using snakeviz?

[etsmith14]

Thanks for the reply. Parallelizing didn't seem to be the problem (parallel = True works fine now). I ended up creating a new virtual environment to reinstall xarray (v0.18.0) and it now runs in about 0.15s which is what I had expected.