5.3.4 causing internal errors

[dwhswenson]

I'm getting internal errors in pytest 5.3.4 on my test suite. Pinning to 5.3.3 fixes. The first few tests run, and then it raises an assertion error with the following traceback:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 197, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 247, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/_pytest/main.py", line 271, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/_pytest/runner.py", line 83, in pytest_runtest_protocol
INTERNALERROR>     item.ihook.pytest_runtest_logstart(nodeid=item.nodeid, location=item.location)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/_pytest/compat.py", line 420, in __get__
INTERNALERROR>     value = instance.__dict__[self.func.__name__] = self.func(instance)
INTERNALERROR>   File "/home/travis/miniconda3/lib/python3.7/site-packages/_pytest/nodes.py", line 465, in location
INTERNALERROR>     assert isinstance(location[0], py.path.local), location[0]
INTERNALERROR> AssertionError: /home/travis/build/openpathsampling/openpathsampling/openpathsampling/tests/test_bias_function.py

This is running on Linux on Travis-CI. Relevant information are in folds within Travis logs; links below will highlight the right line so you can find it more easily.

• Error (5.3.4):
  • https://travis-ci.org/openpathsampling/openpathsampling/jobs/639609917
  • link to conda list output in that job
  • link to error output in that job
• No error (5.3.3)
  • https://travis-ci.org/openpathsampling/openpathsampling/jobs/639614456
  • link to conda list output in that job

For your convenience, here's the diff of the two conda list outputs (copy-pasted to local files):

$ diff tmp_bad tmp_good
50c50
< pytest                    5.3.4                    pypi_0    pypi
---
> pytest                    5.3.3                    pypi_0    pypi

Here's the diff between the two commits:

$ git --no-pager diff HEAD^
diff --git a/devtools/minimal_testing.txt b/devtools/minimal_testing.txt
index c3f4152f..44bbce47 100644
--- a/devtools/minimal_testing.txt
+++ b/devtools/minimal_testing.txt
@@ -1,4 +1,4 @@
 nose
-pytest
+pytest==5.3.3
 pytest-cov
 coveralls

I'm afraid that a minimal example might take me a while, (it's 8:30pm and I want to post this before I leave the office) so I'm leaving that stage out. I hope this is still enough for you to work with!

• a detailed description of the bug or suggestion
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

[nicoddemus]

Thanks @dwhswenson for the detailed report, and sorry about the regression.

That assert doesn't seem to be correct (introduced in 930a158), we should avoid using assert to check public APIs, IMHO.

@dwhswenson do you have some form of custom plugin which is creating pytest.Items? I see you only have pytest-cov installed, so I doubt that is the cause. I was taking a look at the suite, but is quite massive and I have to leave now. I will come back later and can take another look though.

[dwhswenson]

@dwhswenson do you have some form of custom plugin which is creating pytest.Items?

Not that I know of. I don't think our test suite has any custom pytest-based code beyond basic XUnit fixtures and mark.parametrize. These particular errors came from our minimal test job (which doesn't test integrations with optional packages). The entire installation process is: (1) install conda to manage Python versions (this was with 3.7); (2) use pip within conda to install the packages in these files: minimal.txt; minimal_testing.txt. So unless something else came along with one of those packages, there shouldn't be anything.

One thing we do have is lots of legacy nose-based code. Since pytest runs these without trouble, we just keep the nose imports around. And there are weird cases where the same function may include both a nose-like raise SkipTest as well as pytest.importorskip, because those skips were added at different times.

Looking a little more closely, it looks like the error is occurring on the first test that uses the @raises decorator from nose. That makes a hypothesis that I might be able to test out a little later.

[bluetech]

Thanks for reporting @dwhswenson

I was able to reproduce it. PR #6518 should fix it. It doesn't add a regression test, but it's just a revert so can be deferred to later.

Sorry for the hassle @nicoddemus!

[nicoddemus]

Sorry for the hassle @nicoddemus!

Not at all, please. Thanks for the quick response!

[nicoddemus]

Looking a little more closely, it looks like the error is occurring on the first test that uses the @raises decorator from nose. That makes a hypothesis that I might be able to test out a little later.

Thanks @dwhswenson, nailed it! This reproduces the issue:

from nose.tools import raises

@raises(RuntimeError)
def test_fail_without_tcp():
    raise RuntimeError