Dask errors in CI logs

[scottyhq]

Notebooks that create dask clusters (LocalCluster) when executed by jupyter-book build . end up with big tracebacks and end up leaving the dask-worker-space directory... not sure what causes this b/c the notebook execution actually is successful... so it seems like some sort of problem closing the dask cluster:

updating environment: executing outdated notebooks... Executing: /Users/scott/GitHub/xarray-contrib/xarray-tutorial/advanced/xarray_and_dask.ipynb
2022-06-16 11:12:08,966 - distributed.nanny - ERROR - Worker process died unexpectedly
Exception in thread Nanny stop queue watch:
Traceback (most recent call last):
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
2022-06-16 11:12:08,966 - distributed.nanny - ERROR - Worker process died unexpectedly
Exception in thread Nanny stop queue watch:
Traceback (most recent call last):
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 946, in run
2022-06-16 11:12:08,966 - distributed.nanny - ERROR - Worker process died unexpectedly
Exception in thread Nanny stop queue watch:
Traceback (most recent call last):
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/site-packages/distributed/nanny.py", line 846, in watch_stop_q
2022-06-16 11:12:08,967 - distributed.nanny - ERROR - Worker process died unexpectedly
Exception in thread Nanny stop queue watch:
    self._target(*self._args, **self._kwargs)
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/site-packages/distributed/nanny.py", line 846, in watch_stop_q
Traceback (most recent call last):
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 946, in run
    child_stop_q.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/queues.py", line 143, in close
    child_stop_q.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/queues.py", line 143, in close
    self._reader.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 182, in close
    self._target(*self._args, **self._kwargs)
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/site-packages/distributed/nanny.py", line 846, in watch_stop_q
    self.run()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/threading.py", line 946, in run
    self._reader.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 182, in close
    self._close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 366, in _close
    self._close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 366, in _close
    child_stop_q.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/queues.py", line 143, in close
    self._target(*self._args, **self._kwargs)
    _close(self._handle)
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/site-packages/distributed/nanny.py", line 846, in watch_stop_q
OSError: [Errno 9] Bad file descriptor
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor
    self._reader.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 182, in close
    self._close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 366, in _close
    child_stop_q.close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/queues.py", line 143, in close
    _close(self._handle)
    self._reader.close()
OSError: [Errno 9] Bad file descriptor
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 182, in close
    self._close()
  File "/Users/scott/miniconda3/envs/xarray-tutorial/lib/python3.10/multiprocessing/connection.py", line 366, in _close
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor
2022-06-16 11:12:11,148 - distributed.diskutils - INFO - Found stale lock file and directory '/Users/scott/GitHub/xarray-contrib/xarray-tutorial/advanced/dask-worker-space/worker-iqoqflqq', purging
2022-06-16 11:12:11,149 - distributed.diskutils - INFO - Found stale lock file and directory '/Users/scott/GitHub/xarray-contrib/xarray-tutorial/advanced/dask-worker-space/worker-bs9vej_y', purging
2022-06-16 11:12:11,149 - distributed.diskutils - INFO - Found stale lock file and directory '/Users/scott/GitHub/xarray-contrib/xarray-tutorial/advanced/dask-worker-space/worker-asihamol', purging
Execution Succeeded: /Users/scott/GitHub/xarray-contrib/xarray-tutorial/advanced/xarray_and_dask.ipynb
done

[andersy005]

Not 💯 sure this will help but we could set the temporary-directory to an ephemeral location that gets cleanup after the process exit via

In [25]: import dask

In [26]: import tempfile

In [27]: dask.config.set({'temporary-directory': tempfile.mkdtemp()})
Out[27]: <dask.config.set at 0x7f36a46c4d90>

[andersy005]

Also, do you get this behavior when using Python 3.9? I recall seeing some reported issues with dask+multiprocessing in Python 3.10

[scottyhq]

Just did a quick test with client = Client(processes=False) on Python 3.10 and no problem, so I suggest we switch to that, which should be better anyways for the current xarray_and_dask.ipynb doing basic calculations.

I also often use client = Client(local_directory='/tmp') for the worker scratch space...

[dcherian]

^ Sounds good to me.