pytest 4.6.0 breaks matplotlib test gathering with `TypeError: cannot unpack non-iterable NoneType object`

[jkseppan]

This build: https://travis-ci.org/matplotlib/matplotlib/jobs/540086010 passed fine, with pytest 4.5.0. The next build from the same code, initiated via cron and using pytest 4.6.0: https://travis-ci.org/matplotlib/matplotlib/jobs/540212001 failed with

Calling pytest with the following arguments: -raR --maxfail=50 --timeout=300 --durations=25 --cov-report= --cov=lib -n 2 --log-level=DEBUG
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.6.0, py-1.7.0, pluggy-0.12.0
rootdir: /home/travis/build/matplotlib/matplotlib, inifile: pytest.ini, testpaths: lib
plugins: forked-1.0.2, cov-2.7.1, rerunfailures-7.0, xdist-1.28.0, timeout-1.3.3, faulthandler-1.6.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
gw0 I / gw1 I
gw0 [5784] / gw1 [5784]
...
_______ ERROR collecting lib/matplotlib/tests/test_constrainedlayout.py ________
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:963: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:902: in _find_spec
    ???
<frozen importlib._bootstrap>:876: in _find_spec_legacy
    ???
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:169: in find_module
    source_stat, co = _rewrite_test(self.config, fn_pypath)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:404: in _rewrite_test
    rewrite_asserts(tree, fn, config)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:453: in rewrite_asserts
    AssertionRewriter(module_path, config).run(mod)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:701: in run
    new.extend(self.visit(child))
/opt/python/3.7.1/lib/python3.7/ast.py:262: in visit
    return visitor(node)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:828: in visit_Assert
    top_condition, explanation = self.visit(assert_.test)
E   TypeError: cannot unpack non-iterable NoneType object
_______ ERROR collecting lib/matplotlib/tests/test_constrainedlayout.py ________
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:963: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:902: in _find_spec
    ???
<frozen importlib._bootstrap>:876: in _find_spec_legacy
    ???
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:169: in find_module
    source_stat, co = _rewrite_test(self.config, fn_pypath)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:404: in _rewrite_test
    rewrite_asserts(tree, fn, config)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:453: in rewrite_asserts
    AssertionRewriter(module_path, config).run(mod)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:701: in run
    new.extend(self.visit(child))
/opt/python/3.7.1/lib/python3.7/ast.py:262: in visit
    return visitor(node)
../../../virtualenv/python3.7.1/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:828: in visit_Assert
    top_condition, explanation = self.visit(assert_.test)
E   TypeError: cannot unpack non-iterable NoneType object
____________ ERROR collecting lib/matplotlib/tests/test_mathtext.py ____________
In test_mathtext_rendering: 1 parameter sets specified, with different number of ids: 83

The operating system is Ubuntu 16.04.6 LTS (xenial).

I don't know how to unpack the line numbers for the breaking tests from the above, but test_constrainedlayout.py has the following asserts:

    assert all(ax.get_position().extents == ax2.get_position().extents)
    assert all(ax.get_position().extents == ax2.get_position().extents)
    np.testing.assert_allclose(extents0, extents1)
    np.testing.assert_allclose(extents0, extents1)

In addition, there are multiple tests using matplotlib's @image_comparison decorator, but there are so many of those all over the codebase that I think then we'd see much more failures.

• a detailed description of the bug or suggestion
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

[RonnyPfannschmidt]

i believe this is a flaw in the "any unrolling" see #5103

[slarse]

@RonnyPfannschmidt I think that's correct, I battled this problem yesterday and managed to reproduce it with the test case below (or any test case including all)

def test_all():
    assert all([])

Works fine in 4.5.0, breaks in 4.6.0

[RonnyPfannschmidt]

@slarse thanks for providing a minimal reproducer

the triggering code in matplotlib is https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/tests/test_constrainedlayout.py#L300