Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[P2P] merge_unpack does not reschedule on ShuffleClosedError #8660

Open
hendrikmakait opened this issue May 30, 2024 · 0 comments
Open

[P2P] merge_unpack does not reschedule on ShuffleClosedError #8660

hendrikmakait opened this issue May 30, 2024 · 0 comments
Labels
bug Something is broken shuffle

Comments

@hendrikmakait
Copy link
Member

I just noticed that merge_unpack (https://github.com/hendrikmakait/distributed/blob/9fae5dacf4d2cfd5c659e472b0a3ef307d695863/distributed/shuffle/_merge.py#L163-L196) isn't wrapped in handle_unpack_errors like shuffle_unpack (https://github.com/hendrikmakait/distributed/blob/9fae5dacf4d2cfd5c659e472b0a3ef307d695863/distributed/shuffle/_shuffle.py#L94-L101).

This could cause P2P merging not to restart in some edge cases. (I'm not sure which ones, but there should be tests for P2P shuffling that will start to fail if we remove handle_unpack_errors.)

@hendrikmakait hendrikmakait added bug Something is broken shuffle labels May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is broken shuffle
Projects
None yet
Development

No branches or pull requests

1 participant