Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#731] feat(spark): Make blockid layout configurable for Spark clients #1528

Merged
merged 29 commits into from
Mar 7, 2024
Merged

[#731] feat(spark): Make blockid layout configurable for Spark clients #1528

merged 29 commits into from
Mar 7, 2024

Commits on Feb 29, 2024

  1. Move blockId field bit lengths from Constants into BlockIdLayout 

    Also moves bit logic from BlockId into BlockIdLayout
    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    b76ee2d View commit details
    Browse the repository at this point in the history

  2. Create BlockIdLayout from config, use everywhere

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    7179361 View commit details
    Browse the repository at this point in the history

  3. Use 'Bits' rather than 'Length'

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    9b5cd42 View commit details
    Browse the repository at this point in the history

  4. Add block id config to docs, add spark specific considerations

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    4c8f5f2 View commit details
    Browse the repository at this point in the history

  5. Remove map config type, simplify block id layout config

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    dac4d97 View commit details
    Browse the repository at this point in the history

  6. Fix after rebase

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    e984f58 View commit details
    Browse the repository at this point in the history

  7. Use layout to log block id in ShuffleReadClientImpl

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    fc5820c View commit details
    Browse the repository at this point in the history

  8. Add tests

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    7c137d4 View commit details
    Browse the repository at this point in the history

  9. Make DefaultIdHelper not fall back to default BlockIdLayout

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    757ef79 View commit details
    Browse the repository at this point in the history

  10. Remove BlockIdLayout from ShuffleReadClient builder

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    5ba16c2 View commit details
    Browse the repository at this point in the history

  11. Fix test after rebase

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    59f077a View commit details
    Browse the repository at this point in the history

  12. Remove leftovers, have tez sort WriteBufferManager use layout from conf

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    0a55688 View commit details
    Browse the repository at this point in the history

  13. Fix code style

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    7e3c86a View commit details
    Browse the repository at this point in the history

  14. Move block id bits docs from general client to spark guide

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    c1c2fbb View commit details
    Browse the repository at this point in the history

  15. Fix ShuffleReadClientImplTest

    @EnricoMi
    EnricoMi committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    81a33a3 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. Rename getTaskAttemptId to getTaskAttemptIdForBlockId, fix compile error

    @EnricoMi
    EnricoMi committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    6cf1458 View commit details
    Browse the repository at this point in the history

  2. Fix code styles

    @EnricoMi
    EnricoMi committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    d360754 View commit details
    Browse the repository at this point in the history

  3. Better use of parametrized tests

    @EnricoMi
    EnricoMi committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    6583ce4 View commit details
    Browse the repository at this point in the history

  4. Test coordinator dynamic client conf

    @EnricoMi
    EnricoMi committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    3703c4f View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Log block id where partition used to be logged

    @EnricoMi
    EnricoMi committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    6700f36 View commit details
    Browse the repository at this point in the history

  2. Remove default int values, use DEFAULT instance instead

    @EnricoMi
    EnricoMi committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    ed7b3b2 View commit details
    Browse the repository at this point in the history

  3. Handle and test BlockIdLayout not given in getShuffleResult* GRPC req… 

    …uests
    @EnricoMi
    EnricoMi committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    a419387 View commit details
    Browse the repository at this point in the history

  4. Test RssShuffleDataIterator with various layouts

    @EnricoMi
    EnricoMi committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    f3bae27 View commit details
    Browse the repository at this point in the history

  5. Revert removed empty line

    @EnricoMi
    EnricoMi committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    24b153b View commit details
    Browse the repository at this point in the history

  6. Add more details on carefully configuring block id layout for Spark

    @EnricoMi
    EnricoMi committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    f1fc15d View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Fail fast when registering shuffle with unsupported number of partitions

    @EnricoMi
    EnricoMi committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    2ab119a View commit details
    Browse the repository at this point in the history

  2. Add more errors to spark client guide blockid section

    @EnricoMi
    EnricoMi committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    fe3156b View commit details
    Browse the repository at this point in the history

  3. Move fail-fast test to Spark3

    @EnricoMi
    EnricoMi committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    727129e View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Remove default int values (complements ed7b3b2)

    @EnricoMi
    EnricoMi committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    bf0902d View commit details
    Browse the repository at this point in the history