Support running pytest under WASI

[brettcannon]

What's the problem this feature will solve?

I'm trying to run pytest under the WASI build of CPython, but pytest's extensive use of os.dup() prevents it from working.

If you're curious to reproduce:

1. Grab a WASI build from https://github.com/tiran/cpython-wasm-test/releases
2. Unpack it somewhere
3. From the same directory where you unpacked python.wasm: PIP_REQUIRE_VIRTUALENV=0 py -3.11 -m pip install --target . --only-binary=:all: --implementation=py --abi=none --abi=any pytest (if you don't have the Python Launcher, take out -3.11 and add the pip flag --python_version="3.11")
4. Try running wasmtime run --dir=. python.wasm -- -m pytest -h

It looks like:

pytest/src/_pytest/capture.py

Line 381 in 03b1994

self.targetfd_save = os.dup(targetfd)

is the first instance of os.dup() you run into, but even with --capture=no/s, you still run into:

pytest/src/_pytest/faulthandler.py

Line 29 in 61cfaac

stderr_fd_copy = os.dup(get_stderr_fileno())

Describe the solution you'd like

I would love to be able to provide instructions to projects on how to run pytest under WASI to see if their project is compatible.

Alternative Solutions

Additional context

[Zac-HD]

I think webassembly support should be a high priority for Pytest, because it would be so helpful for other packages and maintainers trying to support Python in the browser. I hit similar issues in my Hypothesis Ghostwriter demo, and had to collect and call test functions by hand 😰

I tend to think about this in three stages:

1. Experimental: early adopters play around with Pytest-on-WASI, we accept bugfix PRs for wasm/wasi support, etc. No promises of support, but contributors can get some happy paths working end-to-end. Clear instructions for how early adoptors can contribute are essential - the notes above are a good start but I'd like to make it even easier.
2. Stabilization: once it's practical to test against WASI in CI, we should start doing that. Explicitly document (beta) wasm support, and note on any wasm-incompatible features. Avoid regressions, fix user bug reports, etc.
3. Maintainance: when there's a solid core of regular happy users and bug reports have slowed right down, document as stable. Ensure that not-on-wasm code raises clear and explanatory errors. Continue fixing bug reports, improving features, etc.

[RonnyPfannschmidt]

Is there any way to add some kind of wasi pipeline that gives output to our ci?

Same would go for a headless call to it that gives us the output

Currently I'm under the impression that none of the pytest maintainers has familiarity with wasi

[The-Compiler]

For what it's worth, you should be able to run with -p no:faulthandler to disable that part too.

[RonnyPfannschmidt]

lets add a gh pipeline that runs pytests own testsuite on wasi as a starting point, then continue from there

[brettcannon]

1. the notes above are a good start but I'd like to make it even easier.

This actually came about because I want to document for folks how to test their own stuff at https://github.com/python/cpython/tree/main/Tools/wasm for those willing to be bleeding edge with this sort of thing. My hope was to show a pytest example, but I may have to start with unittest.

Is there any way to add some kind of wasi pipeline that gives output to our ci?

Probably, but it might be a bit janky to start. My hope is we will start releasing official WASI builds of CPython for 3.12 once WASI support reaches tier 2 (I'm already working towards this). So I can also totally understand wanting to wait until that occurs before trying to tackle this in any official capacity.

My main reason for opening this issue was as an FYI for you all and to have an issue available for anyone else who may bump into this while trying stuff out, but I do appreciate that people seemed interesting in seeing if there's a way to make this work somehow!

[brettcannon]

Got the help output!

wasmtime run --dir=. --dir=/dev --allow-precompiled python.cwasm -- -m pytest -s -p no:faulthandler -h

[brettcannon]

I also managed to get the test suite for packaging working with the same set of flags (plus a bunch of massaging of a zip file of the latest release 😅)!