Failing pytester tests with features branch

[The-Compiler]

@nicoddemus Let's continue the discussion from #2945 here, as this probably warrants some deeper investigation.

When trying to run my tests (after some patching), I get various failures in my test_check_coverage.py which runs pytest via pytester and produces a coverage.xml via pytest-cov which it then reads.

Here's how it fails:

___________________________________________________________________________________________________________________________________________________ test_tested_no_branches ___________________________________________________________________________________________________________________________________________________

self = <module 'py.error'>, func = <built-in function open>, args = ('/tmp/pytest-of-florian/pytest-229/test_tested_no_branches0/coverage.xml', 'r'), kwargs = {'encoding': 'utf-8'}, __tracebackhide__ = False, cls = <class 'py.error.ENOENT'>, value = FileNotFoundError(2, 'No such file or directory')
tb = <traceback object at 0x7f5bd063f588>, errno = 2

    def checked_call(self, func, *args, **kwargs):
        """ call a function and raise an errno-exception if applicable. """
        __tracebackhide__ = True
        try:
>           return func(*args, **kwargs)
E           FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-florian/pytest-229/test_tested_no_branches0/coverage.xml'

/home/florian/proj/qutebrowser/git/.tox/py36/lib/python3.6/site-packages/py/_error.py:66: FileNotFoundError

During handling of the above exception, another exception occurred:

covtest = <test_check_coverage.CovtestHelper object at 0x7f5bff541cf8>

    def test_tested_no_branches(covtest):
        covtest.makefile("""
            def func():
                pass
        """)
        covtest.run()
>       assert covtest.check() == []

/home/florian/proj/qutebrowser/git/tests/unit/scripts/test_check_coverage.py:103: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/florian/proj/qutebrowser/git/tests/unit/scripts/test_check_coverage.py:69: in check
    with coverage_file.open(encoding='utf-8') as f:
/home/florian/proj/qutebrowser/git/.tox/py36/lib/python3.6/site-packages/py/_path/local.py:360: in open
    return py.error.checked_call(io.open, self.strpath, mode, encoding=encoding)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <module 'py.error'>, func = <built-in function open>, args = ('/tmp/pytest-of-florian/pytest-229/test_tested_no_branches0/coverage.xml', 'r'), kwargs = {'encoding': 'utf-8'}, __tracebackhide__ = False, cls = <class 'py.error.ENOENT'>, value = FileNotFoundError(2, 'No such file or directory')
tb = <traceback object at 0x7f5bd063f588>, errno = 2

    def checked_call(self, func, *args, **kwargs):
        """ call a function and raise an errno-exception if applicable. """
        __tracebackhide__ = True
        try:
            return func(*args, **kwargs)
        except self.Error:
            raise
        except (OSError, EnvironmentError):
            cls, value, tb = sys.exc_info()
            if not hasattr(value, 'errno'):
                raise
            __tracebackhide__ = False
            errno = value.errno
            try:
                if not isinstance(value, WindowsError):
                    raise NameError
            except NameError:
                # we are not on Windows, or we got a proper OSError
                cls = self._geterrnoclass(errno)
            else:
                try:
                    cls = self._geterrnoclass(_winerrnomap[errno])
                except KeyError:
                    raise value
>           raise cls("%s%r" % (func.__name__, args))
E           py.error.ENOENT: [No such file or directory]: open('/tmp/pytest-of-florian/pytest-229/test_tested_no_branches0/coverage.xml', 'r')

/home/florian/proj/qutebrowser/git/.tox/py36/lib/python3.6/site-packages/py/_error.py:86: ENOENT

I was able to bisect it to 333a9ad but that probably also did some kind of pluggy update?

[The-Compiler]

Bisected to pytest-dev/pluggy@4776698 - cc @tgoodlet

I guess some hook in pytest-cov isn't called anymore or something? Haven't tried reproducing it outside of that test yet.

[The-Compiler]

I tried reproducing it with a minimal test:

pytest_plugins = 'pytester'


def test_foo(testdir):
    testdir.makepyfile(mod="""
    def func():
        pass
    """)
    testdir.makepyfile(test_mod="""
    import mod
    def test_func():
        mod.func()
    """)
    testdir.runpytest('--cov=mod', '--cov-report=xml')
    assert (testdir.tmpdir / 'coverage.xml').exists()

but that passes. I'm out of ideas for now 😟

[The-Compiler]

Ah, I think I understand now! This happens because I use --pythonwarnings error for qutebrowser's testsuite, and the subprocess errors out because of pytest-dev/pluggy#105 since pytest-dev/pluggy@4776698. That of course means that file doesn't exist 😉

[nicoddemus]

Thanks for debugging this @The-Compiler!

I see two options:

1. Fix DeprecationWarning for inspect.getargspec() pluggy#105 / Port to modern inspect pluggy#81 and make a 0.6.1 (or 0.7.0?) release.
2. Leave the deprecation warning in there and users which --pythonwarnings will have to update their "ignore" lists.

What should we do next?

[The-Compiler]

Well, failing all testsuites which use --pythonwarnings with a new release really isn't a nice thing to do 😉

Seeing that there's already a fix at pytest-dev/pluggy#94 I propose merging that and doing a new pluggy release first.

[nicoddemus]

I agree. @RonnyPfannschmidt and @tgoodlet what do you guys think?

[goodboy]

@nicoddemus @The-Compiler sorry, late getting back to you guys.

Yes I think this is a good idea although I think only a patch release should be necessary no?

[nicoddemus]

The next pluggy version is 0.6.0 anyway because we dropped Python 2.6 and Python 3.3 on master.

[goodboy]

0.6.0 is up

[nicoddemus]

Awesome @tgoodlet, thanks!

@The-Compiler would you mind give this another go and make sure this fixes the issue?

[The-Compiler]

Sorry, I forgot about this - seems fine nowadays!