Deprecate UCX-Py backends #120
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rewrite the UCXX/UCX-Py proxy backends added in #116 to deprecate UCX-Py. We have already removed the UCX-Py backend from Distributed in dask/distributed#9105, but while there's no release we cannot rely on that, so the changes in the PR work in the same way we expect when the migration is completed and Distributed has a new release. If distributed-ucxx is not installed a warning and exception are raised telling the user to install it.
The state after this PR is as follows:
protocol="ucx"/protocol="ucxx"should continue working as before, both were already using distributed-ucxxprotocol="ucx-old"(only exists in rapids-dask-dependency added in Add UCXX proxy backend #116) will raise the sameFutureWarningasprotocol="ucx"without RAPIDSprotocol="ucx"/protocol="ucxx"/protocol="ucx-old"all raise aFutureWarningand don’t let users create processes.The final state is expected to be achieved when a new Distributed release is cut and the
__rdd_patch_ucx.pyfile is removed from rapids-dask-dependency, which will be:protocol="ucx"(now transparently uses distributed-ucxx) /protocol="ucxx"(always used distributed-ucxx) should continue working as beforeprotocol="ucx"raises aFutureWarningand doesn’t let users create processesprotocol="ucxx"never existed before distributed-ucxx so it will just fail with a wrong protocol nameSpecifically I've tested the following to confirm what's behavior matches what is described above as the state of this PR (using all
ucx/ucxx/ucx-oldcombinations):dask scheduler --protocol=ucxdask worker ucx://127.0.0.1:8786LocalCluster(protocol="ucx", silence_logs=False)LocalCUDACluster(protocol="ucx", silence_logs=False)