Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 #3259

Closed
wants to merge 46 commits into from
Closed

HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 #3259

wants to merge 46 commits into from

Commits on Nov 10, 2022

  1. [WIP] This is the first version that compiles. 

    Based on: jojochuang@1476163
Added some enhancements
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    dd03fd5 View commit details
    Browse the repository at this point in the history

  2. Update 

    Based on: jojochuang@2d647ce
Added some modifications
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    89b0efd View commit details
    Browse the repository at this point in the history

  3. ShuffleHandler: ch.isOpen() --> ch.isActive()

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    eda8f0f View commit details
    Browse the repository at this point in the history

  4. TestShuffleHandler: Fix mocking in testSendMapCount + replace ch.writ… 

    …e() with ch.writeAndFlush()
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    f657881 View commit details
    Browse the repository at this point in the history

  5. TestShuffleHandler.testMaxConnections: Rewrite test + production code… 

    …: accepted connection handling
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    8efba68 View commit details
    Browse the repository at this point in the history

  6. ShuffleHandler: Fix keepalive test + writing HTTP response properly t… 

    …o channel
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    af59f54 View commit details
    Browse the repository at this point in the history

  7. channel.closeFuture().awaitUninterruptibly() --> channel.close()

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    7a003db View commit details
    Browse the repository at this point in the history

  8. Modify TODOs

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    c8c6a8f View commit details
    Browse the repository at this point in the history

  9. TestShuffleHandler: Add error handling + assertion logic

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    be1c67a View commit details
    Browse the repository at this point in the history

  10. TestShuffleHandler#testSocketKeepAlive: assert HTTP response code + a… 

    …dd bug notes
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    72d6cdf View commit details
    Browse the repository at this point in the history

  11. Fix idle state handling + add test

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    72db0f0 View commit details
    Browse the repository at this point in the history

  12. Add explanation: LastHttpContent.EMPTY_LAST_CONTENT

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    def6ef1 View commit details
    Browse the repository at this point in the history

  13. Fix javac + checkstyle + whitespace issues

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    4c6123e View commit details
    Browse the repository at this point in the history

  14. Attempt to fix tests

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    9a4f963 View commit details
    Browse the repository at this point in the history

  15. code cleanup

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    7547277 View commit details
    Browse the repository at this point in the history

  16. Fix TestShuffleHandler.LoggingHttpResponseEncoder.getExecutingMethodName

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    a1fb09c View commit details
    Browse the repository at this point in the history

  17. Turn back on outbound exception handler in tests

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    368bbdb View commit details
    Browse the repository at this point in the history

  18. TestShuffleHandler: Introduced InputStreamReadResult that stores resp… 

    …onse as string + total bytes read
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    99689d5 View commit details
    Browse the repository at this point in the history

  19. TestShuffleHandler: Use DEFAULT_PORT for all shuffle handler port con… 

    …figs
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    a25aaf3 View commit details
    Browse the repository at this point in the history

  20. Create class: TestExecution: Configure proxy, keep alive connection t… 

    …imeout
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    42138ce View commit details
    Browse the repository at this point in the history

  21. TestExecution: Configure port

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    0618116 View commit details
    Browse the repository at this point in the history

  22. Add logging response encoder to TestShuffleHandler.testMapFileAccess

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    7c9dd4d View commit details
    Browse the repository at this point in the history

  23. TestShuffleHandler.testMapFileAccess: Modify to be able to run it loc… 

    …ally + reproduce jenkins UT failure
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    34bd098 View commit details
    Browse the repository at this point in the history

  24. TestShuffleHandler.testMapFileAccess: Fix in production code

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    23dd755 View commit details
    Browse the repository at this point in the history

  25. TestShuffleHandler.testMapFileAccess: Add back original assertions

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    a25d845 View commit details
    Browse the repository at this point in the history

  26. Turn off debug mode

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    c579a85 View commit details
    Browse the repository at this point in the history

  27. TestShuffleHandler: Stop shufflehandler in all tests, fix debug mode … 

    …issues
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    948aece View commit details
    Browse the repository at this point in the history

  28. TestShuffleHandler.testSocketKeepAlive: Add back assertion

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    56680cf View commit details
    Browse the repository at this point in the history

  29. testReduceFromPartialMem: Add Shuffle IO error assertion to test

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    23a4c03 View commit details
    Browse the repository at this point in the history

  30. LoggingHttpResponseEncoder: Add some new logs

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    9e55965 View commit details
    Browse the repository at this point in the history

  31. Fixed error handling + LastHttpContent 

    - Introduced NettyChannelHelper in ShuffleHandler
- Added Debug / Trace logs to ShuffleHandler
- Fix: Write LastHttpContent.EMPTY_LAST_CONTENT to channel in ShuffleHandler.ReduceMapFileCount#operationComplete
- Fix exception handling + sending HTTP 200 / HTTP 500 responses in ShuffleHandler.Shuffle#channelRead
- Add a flag to control if LoggingHttpResponseEncoder is added to the pipeline (debugging purposes)
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    0f2fb11 View commit details
    Browse the repository at this point in the history

  32. ShuffleHandlerTest fixes + enhancements 

    - Increase URLConnection read timeout / connect timeout when using Debug mode
- Introduce class: ResponseConfig, that stores header + payload data sizes + final HTTP response content-length
- Introduce abstract class: AdditionalMapOutputSenderOperations, that can perform additional operations when sendMap is invoked
- ShuffleHandlerForKeepAliveTests: Enhanced failure control / close channel control
- ShuffleHeaderProvider: Don't compute header on every invocation, cache the size of it
- Fix TestShuffleHandler.HeaderPopulator#populateHeaders: Return full content-length of response, not just the length of the header
- Fix in HttpConnectionHelper#connectToUrlsInternal: Add one headerSize to totalBytesRead.
- Enhancement in HttpConnectionHelper#connectToUrlsInternal: Fail-fast if expected content-length < actual content-length.
- Added new keepalive tests, including: testKeepAliveMultipleMapAttemptIds
- Added new keepalive test with HTTP 400 bad request
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    35d2f04 View commit details
    Browse the repository at this point in the history

  33. Fixed Gergo's comments

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    395f541 View commit details
    Browse the repository at this point in the history

  34. Fix comments from Andras

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    f396a31 View commit details
    Browse the repository at this point in the history

  35. Fix comments from Andras: Use Collections.emptyList() where possible … 

    …in TestShuffleHandler.java
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    77c1095 View commit details
    Browse the repository at this point in the history

  36. Fix comments from Andras: Fix handling the maximum number of connections

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    a8efbf9 View commit details
    Browse the repository at this point in the history

  37. Fix comments from Andras: Replace GlobalEventExecutor.INSTANCE with a… 

    … 5 thread instance of DefaultEventExecutorGroup
    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    2bd9d75 View commit details
    Browse the repository at this point in the history

  38. Code cleanup in TestShuffleHandler.java

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    6145df4 View commit details
    Browse the repository at this point in the history

  39. Attempt to fix shading

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    1994a19 View commit details
    Browse the repository at this point in the history

  40. Empty-Commit

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    741cf52 View commit details
    Browse the repository at this point in the history

  41. Empty-Commit

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    5c6f5b4 View commit details
    Browse the repository at this point in the history

  42. Empty-Commit

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    1690178 View commit details
    Browse the repository at this point in the history

  43. Fix javac issues: assertThat deprecation

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    b9f364d View commit details
    Browse the repository at this point in the history

  44. fix blanks

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    1ae403b View commit details
    Browse the repository at this point in the history

  45. Fix checkstyle

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    4aa8d45 View commit details
    Browse the repository at this point in the history

  46. fix review comments

    @szilard-nemeth
    szilard-nemeth committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    6f36b38 View commit details
    Browse the repository at this point in the history