Run dask's shuffle tests in CI #1
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Just testing this out. This also runs unnecessary tests (ones that don't even use the `shuffle_method` fixture). If we were to really do this, we'd want some way to make pytest search for all tests in dask that use the fixture/have a mark, and import/run them dynamically. Possibly using `pytest.main`?
Just to see how long it takes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Run many tests from dask/dask in distributed's CI against the new P2P shuffle. Currently this runs all of
test_shuffle,test_groupby, andtest_multi.assert_eq/some tests to make them actually use the P2P shuffle)I put the shuffle tests in their own CI job by just extending the
ci1/not ci1matrix. It looks like it takes 9-12min to run them reusing the same cluster for every test. If I removedscope="module", I imagine it would be much, much slower.This isn't how we'd want to implement it for real, just trying this to get a sense of runtimes. If we actually did it, we'd want to collect every test using the
shuffle_methodfixture (this currently both misses some tests, and runs some extraneous tests that don't use the fixture). The pytest-within-pytest incantations did not seem worth figuring out yet, since I'm not sure if we'll want to use this approach at all.So this indicates running the
dask/dasktests in distributed would be possible, though maybe not a good idea.cc @fjetter