Pytest uncaught error when integration tests fail

[alexgibson]

Description

When an integration test fails on stage, we're seeing a bunch of uncaught errors instead of the test failure report: https://github.com/mozilla/bedrock/actions/runs/8262326915/job/22601432494

Possibly related to the Python 3.11 update?

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pytest_parallel/__init__.py", line 93, in run
INTERNALERROR>     run_test(self.session, item, None)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pytest_parallel/__init__.py", line 54, in run_test
INTERNALERROR>     item.ihook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>       ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/assertion/__init__.py", line 175, in pytest_runtest_protocol
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/unittest.py", line 401, in pytest_runtest_protocol
INTERNALERROR>     res = yield
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/faulthandler.py", line 85, in pytest_runtest_protocol
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pytest_rerunfailures.py", line 541, in pytest_runtest_protocol
INTERNALERROR>     reports = runtestprotocol(item, nextitem=nextitem, log=False)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/runner.py", line 134, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/runner.py", line 242, in call_and_report
INTERNALERROR>     report: TestReport = ihook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>       ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 155, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pytest_selenium/pytest_selenium.py", line 280, in pytest_runtest_makereport
INTERNALERROR>     exclude = item.config.getini("selenium_exclude_debug").lower()
INTERNALERROR>     ^^^^^^^^^^^^^^^^^
INTERNALERROR> AttributeError: 'NoneType' object has no attribute 'lower'
INTERNALERROR> 
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/main.py", line 285, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/main.py", line 339, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/_pytest/logging.py", line 806, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/venv/lib/python3.11/site-packages/pytest_parallel/__init__.py", line 334, in pytest_runtestloop
INTERNALERROR>     six.raise_from(exc, err[1])
INTERNALERROR>   File "<string>", line 3, in raise_from
INTERNALERROR> RuntimeError: pytest-parallel got 187 errors, raising the first from Thread-2.

[alexgibson]

/cc @stevejalim @robhudson

[alexgibson]

Another similar stack trace when headless tests failed on prod https://github.com/mozilla/bedrock/actions/runs/8263588521/job/22605363232

[robhudson]

I saw this on my Django 4.2 branch but not on Python 3.11, so I guess it's somewhat random.

I added a commit to my Django 4.2 branch that seemed to fix it:
5f0440c

The problem is pytest-selenium attempts to get the value of the SELENIUM_EXCLUDE_DEBUG setting and in some cases it comes back as None, then it tries to call .lower() on it.

This is odd to me since the default value of this setting is the empty string.
https://github.com/pytest-dev/pytest-selenium/blob/c5be64bc8fffef5f4639a375619b614472f561ab/src/pytest_selenium/pytest_selenium.py#L403

So my thought was by setting any value to it, we avoid it randomly coming back as a None. But it's probably worth a deeper investigation. Perhaps somewhere in the stack an empty string is coerced into a None by other packages. I see pluggy and pytest-parallel also in there.

[robhudson]

Ah, this is a recent change that isn't published yet:
pytest-dev/pytest-selenium@c5be64b#diff-7864af340c72ade8f2e1f097a78f15def6bf4317f8d1acc858548473f66be53a

In the last Python updates I upgraded pytest. I believe the fix is to downgrade it to what we had before. It seems pytest-selenium won't work on pytest>=8 without the above fix.

I'll submit a PR.

[robhudson]

#14318

[robhudson]

Fixed. d20403c