"RuntimeError: NetCDF: Not a valid ID" error when trying to operate on a netCDF file using netCDF4

[ceridwen]

I have some code that writes out a netCDF file,
https://gist.github.com/ceridwen/21f86231e86b72bf18fcc16fcee01548

When I try to do math on this dataset after loading it, though, I get an error:

>>> with xarray.open_dataset('snellman.nc') as dataset:
...   print(dataset)
... 
<xarray.Dataset>
Dimensions:        (event: 91, faction: 20, game: 2643, round: 7)
Coordinates:
  * faction        (faction) |S14 b'acolytes' b'alchemists' b'auren' ...
  * event          (event) |S26 b'action:ACT1' b'action:ACT2' b'action:ACT3' ...
  * game           (game) |S32 b'005Stedda' b'006St3dda' b'00s10' b'0106' ...
  * round          (round) int64 0 1 2 3 4 5 6
Data variables:
    player5        (game) |S17 b'Stedda' b'Muchacho' b'' b'modong' b'' b'' ...
    players_count  (game) int8 5 5 2 5 3 3 4 4 4 4 4 4 5 5 5 5 5 5 5 3 3 4 2 ...
    date           (game) datetime64[ns] 2016-06-07T22:30:47 ...
    player3        (game) |S19 b'Emoht' b'Xrystofer' b'' b'runrun57' ...
    player4        (game) |S20 b'Deryl' b'CarpalTunnel' b'' b'Zak_tak' b'' ...
    player1        (game) |S22 b'ocollard' b'BillyMirmidon' b'00' b'Animax' ...
    scoring_tiles  (game, round) |S6 b'' b'SCORE9' b'SCORE4' b'SCORE2' ...
    events         (game, faction, event, round) float32 nan nan nan nan nan ...
    player2        (game) |S21 b'Mies730' b'Stedda' b'romance90' ...

>>> dataset.mean(dim='round')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.5/site-packages/xarray/core/common.py", line 41, in wrapped_func
    **kwargs)
  File "/home/user/.local/lib/python3.5/site-packages/xarray/core/dataset.py", line 1734, in reduce
    np.issubdtype(var.dtype, np.number) or
  File "/home/user/.local/lib/python3.5/site-packages/xarray/core/variable.py", line 244, in dtype
    return self._data.dtype
  File "/home/user/.local/lib/python3.5/site-packages/xarray/core/utils.py", line 385, in dtype
    return self.array.dtype
  File "/home/user/.local/lib/python3.5/site-packages/xarray/conventions.py", line 436, in dtype
    return np.dtype('S' + str(self.array.shape[-1]))
  File "/home/user/.local/lib/python3.5/site-packages/xarray/core/indexing.py", line 346, in shape
    for size, k in zip(self.array.shape, self.key):
  File "/home/user/.local/lib/python3.5/site-packages/xarray/core/utils.py", line 389, in shape
    return self.array.shape
  File "netCDF4/_netCDF4.pyx", line 3378, in netCDF4._netCDF4.Variable.shape.__get__ (netCDF4/_netCDF4.c:32838)
  File "netCDF4/_netCDF4.pyx", line 3323, in netCDF4._netCDF4.Variable._getdims (netCDF4/_netCDF4.c:31915)
RuntimeError: NetCDF: Not a valid ID

Note that if I generate the dataset without writing it to a file and loading it, the same operation doesn't cause an error:

>>> dataset.mean(dim='round')
<xarray.Dataset>
Dimensions:        (event: 91, faction: 20, game: 2643)
Coordinates:
  * faction        (faction) |S14 b'acolytes' b'alchemists' b'auren' ...
  * event          (event) |S26 b'action:ACT1' b'action:ACT2' b'action:ACT3' ...
  * game           (game) object b'005Stedda' b'006St3dda' b'00s10' b'0106' ...
Data variables:
    player5        (game) |S17 b'Stedda' b'Muchacho' b'' b'modong' b'' b'' ...
    player2        (game) |S21 b'Mies730' b'Stedda' b'romance90' ...
    player4        (game) |S20 b'Deryl' b'CarpalTunnel' b'' b'Zak_tak' b'' ...
    player3        (game) |S19 b'Emoht' b'Xrystofer' b'' b'runrun57' ...
    players_count  (game) int8 5 5 2 5 3 3 4 4 4 4 4 4 5 5 5 5 5 5 5 3 3 4 2 ...
    date           (game) datetime64[ns] 2016-06-07T22:30:47 ...
    player1        (game) |S22 b'ocollard' b'BillyMirmidon' b'00' b'Animax' ...
    events         (game, faction, event) float16 nan nan nan nan nan nan ...

This is with xarray (0.8.2) and netCDF4 (1.2.4). It may be a bug in netCDF rather than xarray itself. See also the crash in #1002, which may or may not be related.

[shoyer]

When you use a with block with an xarray object, it closes any associated netCDF file when you leave the block. When you type dataset.mean(dim='round') the file is already closed.

For interactive use, it's usually preferable to just write dataset = xarray.open_dataset('snellman.nc') and close it explicitly later when/if desired.