Skip to content

Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None #3961

Open
@jessicaaustin

Description

@jessicaaustin

I am testing out code that uses xarray to process netcdf files, in particular to join multiple netcdf files into one along shared dimensions. This was working well, except sometimes when saving the netcdf file the process would hang.

I was able to whittle it down to this simple example: https://github.com/jessicaaustin/xarray_netcdf_hanging_issue

This is the code snippet at the core of the example:

 # If you set lock=False then this runs fine every time.
 # Setting lock=None causes it to intermittently hang on mfd.to_netcdf
 with xr.open_mfdataset(['dataset.nc'], combine='by_coords', lock=None) as mfd:
     p = os.path.join('tmp', 'xarray_{}.nc'.format(uuid.uuid4().hex))
     print(f"Writing data to {p}")
     mfd.to_netcdf(p)
     print("complete")

If you run this once, it's typically fine. But run it over and over again in a loop, and it'll eventually hang on mfd.to_netcdf. However if I set lock=False then it runs fine every time.

I've seen this with the following combos:

  • xarray=0.14.1
  • dask=2.9.1
  • netcdf4=1.5.3

and

  • xarray=0.15.1
  • dask=2.14.0
  • netcdf4=1.5.3

And I've tried it with different netcdf files and different computers.

Versions

Output of `xr.show_versions()`

INSTALLED VERSIONS

commit: None
python: 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.15.0-20-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.4

xarray: 0.15.1
pandas: 1.0.3
numpy: 1.18.1
scipy: None
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.1.1.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.14.0
distributed: 2.14.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
setuptools: 46.1.3.post20200325
pip: 20.0.2
conda: None
pytest: None
IPython: None
sphinx: None

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions