Description
I am testing out code that uses xarray to process netcdf files, in particular to join multiple netcdf files into one along shared dimensions. This was working well, except sometimes when saving the netcdf file the process would hang.
I was able to whittle it down to this simple example: https://github.com/jessicaaustin/xarray_netcdf_hanging_issue
This is the code snippet at the core of the example:
# If you set lock=False then this runs fine every time.
# Setting lock=None causes it to intermittently hang on mfd.to_netcdf
with xr.open_mfdataset(['dataset.nc'], combine='by_coords', lock=None) as mfd:
p = os.path.join('tmp', 'xarray_{}.nc'.format(uuid.uuid4().hex))
print(f"Writing data to {p}")
mfd.to_netcdf(p)
print("complete")
If you run this once, it's typically fine. But run it over and over again in a loop, and it'll eventually hang on mfd.to_netcdf
. However if I set lock=False
then it runs fine every time.
I've seen this with the following combos:
- xarray=0.14.1
- dask=2.9.1
- netcdf4=1.5.3
and
- xarray=0.15.1
- dask=2.14.0
- netcdf4=1.5.3
And I've tried it with different netcdf files and different computers.
Versions
Output of `xr.show_versions()`
INSTALLED VERSIONS
commit: None
python: 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.15.0-20-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.4
xarray: 0.15.1
pandas: 1.0.3
numpy: 1.18.1
scipy: None
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.1.1.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.14.0
distributed: 2.14.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
setuptools: 46.1.3.post20200325
pip: 20.0.2
conda: None
pytest: None
IPython: None
sphinx: None