Python unit tests using multiprocessing fail/hang on macOS + Python 3.8

[Hexcles]

I'm seeing these (new?) failures in both tools/ and tools/wpt tests on Azure:
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=52012&view=logs&jobId=1d360f5f-cec3-58ac-12a7-7f135bf134d9&j=1d360f5f-cec3-58ac-12a7-7f135bf134d9&t=a042d866-5c9d-573a-d81c-85b65eaf66e4
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=52012&view=logs&jobId=1d360f5f-cec3-58ac-12a7-7f135bf134d9&j=8018c8f5-0b0f-5870-6bc1-b62b0370009f&t=ee261702-2b78-51d0-d8fc-5ff4bfe2de49

The errors are a bit cryptic, but similar. Among them, this looks the most suspicious:

ImportError: cannot import name 'console_main' from 'pytest' (/Users/runner/work/1/s/tools/third_party/pytest/src/pytest.py)

[Hexcles]

(I'm trying to repro locally but will first need to get py3.8.)

[Hexcles]

I can confirm locally this only happens in Python 3.8.

[Hexcles]

More digging suggests the root cause is due to a change in the default multiprocessing start method:
https://docs.python.org/3.8/library/multiprocessing.html#contexts-and-start-methods

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

which has some additional requirements: https://docs.python.org/3.8/library/multiprocessing.html#the-spawn-and-forkserver-start-methods

The ImportError seems to suggest that the pytest entrypoint doesn't satisfy one of the requirements, and it looks like we're not alone:
scipy/scipy#11835
pytest-dev/pytest#958

I've tried the workaround used by scipy, importing multiprocessing (without using it) at various places, but with no luck.

I'm going to disable the affected tests for now.

[gsnedders]

Oddly, the spawn server was already in use on Windows, so it's surprising this wasn't already failing on Windows.

[LukeZielinski]

Looks like this has been mitigated...should we downgrade from roadmap to backlog, or are we expecting to fix any time soon?

[Hexcles]

I think we should keep it at roadmap given that we're moving to py3-only and we should start fixing & unskipping tests in PY3.

[ziransun]

I have some findings and would appreciate your thoughts -

ImportError: cannot import name 'console_main' from 'pytest' (/Users/runner/work/1/s/tools/third_party/pytest/src/pytest.py)

"console_main()" was introduced to pytest here, about six months ago. Source code in tools/third_party/pytest/ is older and doesn't have console_main().

In a local run with tox command, for Py38 it creates wpt/tools/.tox/py38/bin/pytest which calls console_main as follows:

from pytest import console_main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])

replace console_main with main, we have

from pytest import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Tests that hang due "ImportError: cannot import name 'console_main' from 'pytest' (/Users/runner/work/1/s/tools/third_party/pytest/src/pytest.py)" now seem running fine in MacOS with python 3.8

There are a couple of questions here:
[1] Any reason pytest is looking for library in tools/third_party/pytest/ rather than in tools/.tox/py38/bin/pytest/lib? Is it an expected behaviour?
[2] Does it make sense to update tools/third_party/pytest to the latest version?

Thanks!

[Hexcles]

I took a deeper look and this is a bit convoluted.

First of all, it's important to note that the ImportError comes from Stash which runs in a separate process via multiprocessing.manager instead of the main process (pytest started by tox). tox always starts the latest version of pytest in its venv, without any issue. However, in Stash, multiprocessing tries to spawn a new process, using the same entry point as the original process (i.e. pytest) but with sys.path now updated by the code under test (wptserve) to include tools/third_party (by importing localpaths). Now the new process would find tools/third_party/pytest instead of the pytest in tox's venv.

Regarding the solution, I think upgrading tools/third_party/pytest should fix the issue. Note that we'd need a recent version (>=6.2.0) which does not support Python 2. Also #26132 is a blocker. Alternatively, we might try appending to sys.path instead of prepending in localpaths.py:

wpt/tools/localpaths.py

Line 16 in 01922bd

sys.path.insert(0, os.path.join(here, "third_party", "pytest"))

But I'm sure that has its own set of issues (maybe @jgraham would know).

[gsnedders]

This is kinda why I'd like us to end up with us importing third_party.pytest or similar rather than the status-quo if we want to be ensuring things are using our vendored copies.

[Hexcles]

Apparently I got confused by another issue and thought there was no pytest version that supported both py2 & 3.

There is: https://docs.pytest.org/en/stable/py27-py34-deprecation.html

So a workaround is to pin to 4.6 both in tools/third_party and in tox.ini. Let me give it a try.