Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically restart P2P shuffles when output worker leaves #7970

Merged
merged 88 commits into from
Jul 24, 2023
Merged

Automatically restart P2P shuffles when output worker leaves #7970

merged 88 commits into from
Jul 24, 2023

Commits on Jul 5, 2023

  1. Adjust tests

    @hendrikmakait
    hendrikmakait committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    6658a65 View commit details
    Browse the repository at this point in the history

  2. Restart shuffle

    @hendrikmakait
    hendrikmakait committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    4e7b425 View commit details
    Browse the repository at this point in the history

  3. Remove erred_shuffles

    @hendrikmakait
    hendrikmakait committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    1cfa0d3 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. Fix failures on unpack

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    fe79aa3 View commit details
    Browse the repository at this point in the history

  2. transitions

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    853d953 View commit details
    Browse the repository at this point in the history

  3. Revert "transitions" 

    This reverts commit 853d953.
    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    e058234 View commit details
    Browse the repository at this point in the history

  4. Adjust tests

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    8510c06 View commit details
    Browse the repository at this point in the history

  5. Recommendations

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    6138fc6 View commit details
    Browse the repository at this point in the history

  6. stimulus_id in remove_worker and transition

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    0e7b051 View commit details
    Browse the repository at this point in the history

  7. Use stimulus_id as transaction

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    8e92b85 View commit details
    Browse the repository at this point in the history

  8. Tests

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    3be333e View commit details
    Browse the repository at this point in the history

  9. to kwarg

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    8412d2c View commit details
    Browse the repository at this point in the history

  10. kwarg

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    ba67405 View commit details
    Browse the repository at this point in the history

  11. warning

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    9d804dc View commit details
    Browse the repository at this point in the history

  12. forgiveness

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    0fd66cd View commit details
    Browse the repository at this point in the history

  13. forgiveness

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    4f456e9 View commit details
    Browse the repository at this point in the history

  14. forgiveness

    @hendrikmakait
    hendrikmakait committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    8f02f8d View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Minor

    @hendrikmakait
    hendrikmakait committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    29405c6 View commit details
    Browse the repository at this point in the history

  2. Move warning

    @hendrikmakait
    hendrikmakait committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    476f132 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    d34917e View commit details
    Browse the repository at this point in the history

  4. Minor

    @hendrikmakait
    hendrikmakait committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    69f38f8 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    337dc72 View commit details
    Browse the repository at this point in the history

  6. Closing

    @hendrikmakait
    hendrikmakait committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    2a18033 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. Improved error messages

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    85c7ab2 View commit details
    Browse the repository at this point in the history

  2. Adjust tests

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    263a75b View commit details
    Browse the repository at this point in the history

  3. Fix

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    266732e View commit details
    Browse the repository at this point in the history

  4. Improved

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    8fc28eb View commit details
    Browse the repository at this point in the history

  5. Fix

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    fef0a94 View commit details
    Browse the repository at this point in the history

  6. Minor

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    2d2923f View commit details
    Browse the repository at this point in the history

  7. Merge branch 'stimuli-in-plugin-hooks' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    9936857 View commit details
    Browse the repository at this point in the history

  8. Minor

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    fd00098 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'improved-p2p-errors' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    1b7d055 View commit details
    Browse the repository at this point in the history

  10. Apply suggestions from code review 

    Co-authored-by: Lawrence Mitchell <wence@gmx.li>
    @hendrikmakait @wence-
    hendrikmakait and wence- authored Jul 10, 2023
    Configuration menu
    Copy the full SHA
    88f11a4 View commit details
    Browse the repository at this point in the history

  11. Suggestions from code review

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    a70d1ec View commit details
    Browse the repository at this point in the history

  12. Merge branch 'improved-p2p-errors' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    976ddb3 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'main' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    559075b View commit details
    Browse the repository at this point in the history

  14. Rename and use teardown

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    c70b4cb View commit details
    Browse the repository at this point in the history

  15. Renaming

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    9790360 View commit details
    Browse the repository at this point in the history

  16. Test shutdown

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    25a5cc3 View commit details
    Browse the repository at this point in the history

  17. Check for teardown

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    593fa02 View commit details
    Browse the repository at this point in the history

  18. Fix tests

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    f4b5f1c View commit details
    Browse the repository at this point in the history

  19. Fix tests

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    e0d0174 View commit details
    Browse the repository at this point in the history

  20. Merge branch 'shuffle-shutdown' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    ff711e2 View commit details
    Browse the repository at this point in the history

  21. Fix

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    31929ab View commit details
    Browse the repository at this point in the history

  22. Fix check

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    f83bc40 View commit details
    Browse the repository at this point in the history

  23. Merge branch 'shuffle-shutdown' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    6059f91 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. Default timeout on tests

    @hendrikmakait
    hendrikmakait committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    dca742e View commit details
    Browse the repository at this point in the history

  2. Ensure that worker is running before letting it participate

    @hendrikmakait
    hendrikmakait committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    47eacfc View commit details
    Browse the repository at this point in the history

  3. Notify in more places

    @hendrikmakait
    hendrikmakait committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    f544611 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. Avoid problems with queueing

    @hendrikmakait
    hendrikmakait committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    a357e38 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2023

  1. Cleanup

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    34062b1 View commit details
    Browse the repository at this point in the history

  2. Code review

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    bbfa1b9 View commit details
    Browse the repository at this point in the history

  3. kwargs in remove_worker

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    10126ff View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into stimuli-in-plugin-hooks

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    1482153 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'stimuli-in-plugin-hooks' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    36143fe View commit details
    Browse the repository at this point in the history

  6. Fix cleanup

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    ed69caf View commit details
    Browse the repository at this point in the history

  7. Fix test

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    bd74bf2 View commit details
    Browse the repository at this point in the history

  8. Scheduler state cleanup

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    74edf2e View commit details
    Browse the repository at this point in the history

  9. Revert change

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    391042e View commit details
    Browse the repository at this point in the history

  10. Improve get_or_create

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    788f8a4 View commit details
    Browse the repository at this point in the history

  11. Prohibit non-processing tasks from creating a shuffle state

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    c51e39e View commit details
    Browse the repository at this point in the history

  12. Minor

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    cf2ac3c View commit details
    Browse the repository at this point in the history

  13. Remove superfluous test

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    6b999b0 View commit details
    Browse the repository at this point in the history

  14. Cleanup

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    396f1c2 View commit details
    Browse the repository at this point in the history

  15. Additional tests

    @hendrikmakait
    hendrikmakait committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    5070674 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. no cover

    @hendrikmakait
    hendrikmakait committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    938eab1 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. docstrings

    @hendrikmakait
    hendrikmakait committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    6cfe848 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. Update distributed/shuffle/_scheduler_plugin.py 

    Co-authored-by: Lawrence Mitchell <wence@gmx.li>
    @hendrikmakait @wence-
    hendrikmakait and wence- authored Jul 18, 2023
    Configuration menu
    Copy the full SHA
    6c1be53 View commit details
    Browse the repository at this point in the history

  2. Dataclass init

    @hendrikmakait
    hendrikmakait committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    f904377 View commit details
    Browse the repository at this point in the history

  3. Update distributed/shuffle/_scheduler_plugin.py 

    Co-authored-by: Lawrence Mitchell <wence@gmx.li>
    @hendrikmakait @wence-
    hendrikmakait and wence- authored Jul 18, 2023
    Configuration menu
    Copy the full SHA
    160d055 View commit details
    Browse the repository at this point in the history

  4. Review feedback

    @hendrikmakait
    hendrikmakait committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    71778e6 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. Merge branch 'main' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    00df2e8 View commit details
    Browse the repository at this point in the history

  2. Remove notify_all

    @hendrikmakait
    hendrikmakait committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    f1ccd63 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2023

  1. Remove __eq__

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    3ac439c View commit details
    Browse the repository at this point in the history

  2. walrussin'

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    d72596b View commit details
    Browse the repository at this point in the history

  3. Condition

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    f14c71b View commit details
    Browse the repository at this point in the history

  4. Docstring

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    90eb9ea View commit details
    Browse the repository at this point in the history

  5. cond

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    4b4ab50 View commit details
    Browse the repository at this point in the history

  6. Simplify restart

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    dd7ac3e View commit details
    Browse the repository at this point in the history

  7. raise if invariant broken

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    9459019 View commit details
    Browse the repository at this point in the history

  8. Additional test

    @hendrikmakait
    hendrikmakait committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    3213ef5 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2023

  1. Fix race

    @hendrikmakait
    hendrikmakait committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    e4be94f View commit details
    Browse the repository at this point in the history

  2. Fix test

    @hendrikmakait
    hendrikmakait committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    c1ef0be View commit details
    Browse the repository at this point in the history

  3. No reproducer

    @hendrikmakait
    hendrikmakait committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    434d5a3 View commit details
    Browse the repository at this point in the history

  4. doc and ignore

    @hendrikmakait
    hendrikmakait committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    1f0ea84 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. Merge branch 'main' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    9e7c2e8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into restart-p2p-shuffle

    @hendrikmakait
    hendrikmakait committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    36a2c94 View commit details
    Browse the repository at this point in the history

  3. [skip-caching]

    @hendrikmakait
    hendrikmakait committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    5764f45 View commit details
    Browse the repository at this point in the history