-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 #3259
Commits on Nov 10, 2022
-
[WIP] This is the first version that compiles.
Based on: jojochuang@1476163 Added some enhancements
Configuration menu - View commit details
-
Copy full SHA for dd03fd5 - Browse repository at this point
Copy the full SHA dd03fd5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 89b0efd - Browse repository at this point
Copy the full SHA 89b0efdView commit details -
Configuration menu - View commit details
-
Copy full SHA for eda8f0f - Browse repository at this point
Copy the full SHA eda8f0fView commit details -
TestShuffleHandler: Fix mocking in testSendMapCount + replace ch.writ…
…e() with ch.writeAndFlush()
Configuration menu - View commit details
-
Copy full SHA for f657881 - Browse repository at this point
Copy the full SHA f657881View commit details -
TestShuffleHandler.testMaxConnections: Rewrite test + production code…
…: accepted connection handling
Configuration menu - View commit details
-
Copy full SHA for 8efba68 - Browse repository at this point
Copy the full SHA 8efba68View commit details -
Configuration menu - View commit details
-
Copy full SHA for af59f54 - Browse repository at this point
Copy the full SHA af59f54View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7a003db - Browse repository at this point
Copy the full SHA 7a003dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for c8c6a8f - Browse repository at this point
Copy the full SHA c8c6a8fView commit details -
Configuration menu - View commit details
-
Copy full SHA for be1c67a - Browse repository at this point
Copy the full SHA be1c67aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 72d6cdf - Browse repository at this point
Copy the full SHA 72d6cdfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 72db0f0 - Browse repository at this point
Copy the full SHA 72db0f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for def6ef1 - Browse repository at this point
Copy the full SHA def6ef1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c6123e - Browse repository at this point
Copy the full SHA 4c6123eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a4f963 - Browse repository at this point
Copy the full SHA 9a4f963View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7547277 - Browse repository at this point
Copy the full SHA 7547277View commit details -
Configuration menu - View commit details
-
Copy full SHA for a1fb09c - Browse repository at this point
Copy the full SHA a1fb09cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 368bbdb - Browse repository at this point
Copy the full SHA 368bbdbView commit details -
TestShuffleHandler: Introduced InputStreamReadResult that stores resp…
…onse as string + total bytes read
Configuration menu - View commit details
-
Copy full SHA for 99689d5 - Browse repository at this point
Copy the full SHA 99689d5View commit details -
Configuration menu - View commit details
-
Copy full SHA for a25aaf3 - Browse repository at this point
Copy the full SHA a25aaf3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 42138ce - Browse repository at this point
Copy the full SHA 42138ceView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0618116 - Browse repository at this point
Copy the full SHA 0618116View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7c9dd4d - Browse repository at this point
Copy the full SHA 7c9dd4dView commit details -
TestShuffleHandler.testMapFileAccess: Modify to be able to run it loc…
…ally + reproduce jenkins UT failure
Configuration menu - View commit details
-
Copy full SHA for 34bd098 - Browse repository at this point
Copy the full SHA 34bd098View commit details -
Configuration menu - View commit details
-
Copy full SHA for 23dd755 - Browse repository at this point
Copy the full SHA 23dd755View commit details -
Configuration menu - View commit details
-
Copy full SHA for a25d845 - Browse repository at this point
Copy the full SHA a25d845View commit details -
Configuration menu - View commit details
-
Copy full SHA for c579a85 - Browse repository at this point
Copy the full SHA c579a85View commit details -
Configuration menu - View commit details
-
Copy full SHA for 948aece - Browse repository at this point
Copy the full SHA 948aeceView commit details -
Configuration menu - View commit details
-
Copy full SHA for 56680cf - Browse repository at this point
Copy the full SHA 56680cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 23a4c03 - Browse repository at this point
Copy the full SHA 23a4c03View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e55965 - Browse repository at this point
Copy the full SHA 9e55965View commit details -
Fixed error handling + LastHttpContent
- Introduced NettyChannelHelper in ShuffleHandler - Added Debug / Trace logs to ShuffleHandler - Fix: Write LastHttpContent.EMPTY_LAST_CONTENT to channel in ShuffleHandler.ReduceMapFileCount#operationComplete - Fix exception handling + sending HTTP 200 / HTTP 500 responses in ShuffleHandler.Shuffle#channelRead - Add a flag to control if LoggingHttpResponseEncoder is added to the pipeline (debugging purposes)
Configuration menu - View commit details
-
Copy full SHA for 0f2fb11 - Browse repository at this point
Copy the full SHA 0f2fb11View commit details -
ShuffleHandlerTest fixes + enhancements
- Increase URLConnection read timeout / connect timeout when using Debug mode - Introduce class: ResponseConfig, that stores header + payload data sizes + final HTTP response content-length - Introduce abstract class: AdditionalMapOutputSenderOperations, that can perform additional operations when sendMap is invoked - ShuffleHandlerForKeepAliveTests: Enhanced failure control / close channel control - ShuffleHeaderProvider: Don't compute header on every invocation, cache the size of it - Fix TestShuffleHandler.HeaderPopulator#populateHeaders: Return full content-length of response, not just the length of the header - Fix in HttpConnectionHelper#connectToUrlsInternal: Add one headerSize to totalBytesRead. - Enhancement in HttpConnectionHelper#connectToUrlsInternal: Fail-fast if expected content-length < actual content-length. - Added new keepalive tests, including: testKeepAliveMultipleMapAttemptIds - Added new keepalive test with HTTP 400 bad request
Configuration menu - View commit details
-
Copy full SHA for 35d2f04 - Browse repository at this point
Copy the full SHA 35d2f04View commit details -
Configuration menu - View commit details
-
Copy full SHA for 395f541 - Browse repository at this point
Copy the full SHA 395f541View commit details -
Configuration menu - View commit details
-
Copy full SHA for f396a31 - Browse repository at this point
Copy the full SHA f396a31View commit details -
Fix comments from Andras: Use Collections.emptyList() where possible …
…in TestShuffleHandler.java
Configuration menu - View commit details
-
Copy full SHA for 77c1095 - Browse repository at this point
Copy the full SHA 77c1095View commit details -
Configuration menu - View commit details
-
Copy full SHA for a8efbf9 - Browse repository at this point
Copy the full SHA a8efbf9View commit details -
Fix comments from Andras: Replace GlobalEventExecutor.INSTANCE with a…
… 5 thread instance of DefaultEventExecutorGroup
Configuration menu - View commit details
-
Copy full SHA for 2bd9d75 - Browse repository at this point
Copy the full SHA 2bd9d75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6145df4 - Browse repository at this point
Copy the full SHA 6145df4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1994a19 - Browse repository at this point
Copy the full SHA 1994a19View commit details -
Configuration menu - View commit details
-
Copy full SHA for 741cf52 - Browse repository at this point
Copy the full SHA 741cf52View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c6f5b4 - Browse repository at this point
Copy the full SHA 5c6f5b4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1690178 - Browse repository at this point
Copy the full SHA 1690178View commit details -
Configuration menu - View commit details
-
Copy full SHA for b9f364d - Browse repository at this point
Copy the full SHA b9f364dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1ae403b - Browse repository at this point
Copy the full SHA 1ae403bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4aa8d45 - Browse repository at this point
Copy the full SHA 4aa8d45View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f36b38 - Browse repository at this point
Copy the full SHA 6f36b38View commit details