Spike/pytest xdist

[leshy]

generally development of pytest and xdist seems slow, hacky, not documented and built on bad foundations,
nothing is composable, async, fixtures, forced incremental execution, parallelism work individually but not together in any combination without hacking with weird glue code.

Issues with pytest:

• no async support (hacked for now, forcing sync execution)
• no async fixture support (hacked for now forcing sync)
• even tho we hacked async in - no parallelism anywhere
• some libraries to address some of these things, but everything I looked at by now just took away time with questionable results and incompatibility with everything else

xdist runs a pool of processes and parallelizes pytest

• inefficient as async functions are still run in a sync fashion within processes
• takes away the chance to order tests (Use fixtures to better distribute tests via load scheduling pytest-dev/pytest-xdist#18) - would we want this? I'd like my CRUD tests for example to be separate ordered test cases with side-effects but not sure.
• xdist re-executes module-level fixtures in different processes, slow and can cause issues if fixtures have sideeffects (creating a file)

Suggestion:

a good light testing framework for our needs is less then 200 lines of code, one or two days of work, let's perhaps run away from all this?
we can then have pluggable test parsers depending on the file prefix, "caitest_" files could be parsed by our awesome nice async tester, while "test_" ones are passed to pytest..

for now, instead of xdist we can DIY, parallelize pytest on a per file basis (my spike here,
getting the same performance as XDIST (running our full test stack)),

this allows us to in the future implement our own good testcase parser.

XDIST 0m25-30s
IVAN 0m25-30s
PYTEST 1m0.578s

to run my test runner:

cd tests
python testrunner.py

to run xdist:

pip install pytest-xdist
pytest -n auto tests/

tests for the new parser can look very similar to pytest tests and existing pytest tests that we feel are slow can be ported quickly, if not be compatible off the bat.

not sure what the consequences are in terms of devs using their IDEs to integrate testing?