perf: faster test suite with xdist

[beckermr]

As it says...

[cvanelteren]

The problem is python xdist does not play well with pytest-mpl out of the box.

[cvanelteren]

You could perhaps look at this: matplotlib/pytest-mpl#136 (comment). Ended up not completing it, but it is doable.

[beckermr]

The problem is python xdist does not play well with pytest-mpl out of the box.

What happens? This is something I am happy to look into.

[cvanelteren]

The issue is that the tests need to be collected on a singular thread. The tests are reported correctly, but not put in the correct folder. I was playing around with it in the link above. Was able to get it working but didn't pursue it further to clean it up. Ideally we would merge it with them for native support (and force their hand).

[codecov]

Codecov Report

Attention: Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
ultraplot/tests/test_format.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[beckermr]

So there are multiple bugs related to using xdist here, both due to removing non-failed comparisons and due to writing the report. This will take me longer than I had guessed. :/

[cvanelteren]

I ended up writing s custom html page and just moved the failed test over. But dunno if I still have the code. Will check when I have a computer.

[beckermr]

I tried the code in your issue and it doesn't work with the latest mpl pytest. I plan to look later I am just very busy in the next two weeks.

[cvanelteren]

Couldn't find the code unfortunately :( Thinks I torched the branch.

[beckermr]

I am seeing non-reproducible failures in the main test suite when using xdist in addition to failures in the image comparisons. I think this points to something more fundamentally wrong. So I am going to abandon this PR for now in favor of #213 which gives a 25% reduction in the runtime of the tests without the tricky failures.