NBMAKE INTERNAL ERROR: Exception ignored in socket

[adamjstewart]

Describe the bug

I'm seeing the following error message in CI:

NBMAKE INTERNAL ERROR
Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/jsonschema/exceptions.py", line 134, in _set
    for k, v in kwargs.items():
ResourceWarning: unclosed <socket.socket fd=27, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>

  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/runner.py", line 338, in from_call
    result: Optional[TResult] = func()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/runner.py", line [25](https://github.com/microsoft/torchgeo/runs/8172122269?check_suite_focus=true#step:5:26)9, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line [26](https://github.com/microsoft/torchgeo/runs/8172122269?check_suite_focus=true#step:5:27)5, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 88, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

To Reproduce

1. See https://github.com/microsoft/torchgeo/runs/8172122269?check_suite_focus=true
2. Failing notebook is https://github.com/microsoft/torchgeo/blob/releases/v0.3/docs/tutorials/transforms.ipynb

Expected behavior

I would expect nbmake to either pass all tests, or inform me of what tests may have failed.

Desktop

• OS: Ubuntu 20.04
• Browser: Chrome
• Version: 1.3.4

Additional context

Seems similar to #71 but the error message is slightly different.

[adamjstewart]

I believe the notebook failure is correct, but the error message isn't particularly informative. Replacing %nvidia-smi with !nvidia-smi fixes the notebook on Google Colab, although it may still break in CI where there is no GPU installed. We should still fix the error message if we can.

[alex-treebeard]

Hi Adam,

I can't reproduce this, is it possible this was a transient error relating to some of the Ipython network components?

I have however discovered a bug: It seems like UsageErrors relating to unknown magic commands such as %nvidia-smi halt execution due to the error, yet are reported by nbmake as successful.

[adamjstewart]

Unfortunately this happens every time for me consistently. I'll try to reproduce it locally.

[adamjstewart]

Couldn't reproduce locally. Could be something specific to the CI environment, or could be a difference in dependency versions.

[alex-treebeard]

Strange. Do you know how %nvidia-smi is installed?

I tried that magic command on a colab notebook and found it didn't exist.

Are there any downsides from your workaround of using !nvidia-smi?

[adamjstewart]

There is no %nvidia-smi magic command, you have to use !nvidia-smi instead. The bug isn't that %nvidia-smi doesn't exist, the bug is that nbmake fails with a mysterious error that doesn't help me locate the problem.

[adamjstewart]

FWIW, still seeing this every time we try to test our notebooks: https://github.com/microsoft/torchgeo/actions/runs/4127657580/jobs/7131131990

[alex-treebeard]

thanks for persistently reporting this @adamjstewart

I hit a brick wall last time because I couldn't repro. Are you able to provide a docker image where we can repro this bug?

[adamjstewart]

I'm also still trying to repro locally. So far it only happens for us in CI. I did notice that some of our cells timeout when run locally, so I increased the default timeout period. But I'm still seeing the issue.

is it possible this was a transient error relating to some of the Ipython network components?

Unfortunately this happens every time for me consistently.

I will clarify this and say that the error happens ~100% of the time, but the notebook that errors out keeps changing. In microsoft/torchgeo#1097, the notebook that was failing switched from pretrained_weights.ipynb to trainers.ipynb. It could be that reproducing the issue requires running all notebooks, not just one notebook.

Also note that the error is slightly different than last time. Now we have:

NBMAKE INTERNAL ERROR
Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/nbformat/__init__.py", line 166, in read
    buf = fp.read()
AttributeError: 'str' object has no attribute 'read'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/nbformat/notebooknode.py", line 47, in from_dict
    return NotebookNode({k: from_dict(v) for k, v in d.items()})
ResourceWarning: unclosed <socket.socket fd=23, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>

  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/_pytest/runner.py", line 339, in from_call
    result: Optional[TResult] = func()
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/_pytest/runner.py", line 260, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 88, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

Unfortunately I don't know how to get a full stacktrace so I can't know what this str object is or why it isn't a filehandle.

[adamjstewart]

This is interesting. The error points to an AttributeError triggered here. However, this should be caught by the following line. I'm not sure why it isn't being caught.