Skip to content

HDDS-1805. Implement S3 Initiate MPU request to use Cache and DoubleBuffer. #1108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 20, 2019
Merged

HDDS-1805. Implement S3 Initiate MPU request to use Cache and DoubleBuffer. #1108

merged 1 commit into from
Jul 20, 2019

Conversation

bharatviswa504
Copy link
Contributor

@bharatviswa504 bharatviswa504 commented Jul 16, 2019 •
edited
Loading

First commit is S3 DeleteBucket request.
Posted for initial CI, once HDDS-1795 gets in, will rebase this.

@bharatviswa504 bharatviswa504 requested a review from arp7 July 16, 2019 21:27
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 35 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 11 Maven dependency ordering for branch
+1 mvninstall 454 trunk passed
+1 compile 255 trunk passed
+1 checkstyle 65 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 797 branch has no errors when building and testing our client artifacts.
+1 javadoc 151 trunk passed
0 spotbugs 319 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 507 trunk passed
_ Patch Compile Tests _
0 mvndep 25 Maven dependency ordering for patch
-1 mvninstall 42 hadoop-hdds in the patch failed.
+1 compile 253 the patch passed
+1 cc 253 the patch passed
+1 javac 253 the patch passed
+1 checkstyle 73 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 675 patch has no errors when building and testing our client artifacts.
+1 javadoc 162 the patch passed
+1 findbugs 523 the patch passed
_ Other Tests _
+1 unit 284 hadoop-hdds in the patch passed.
-1 unit 1771 hadoop-ozone in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
6636
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/1/artifact/out/Dockerfile
GITHUB PR #1108
Optional Tests dupname asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle
uname Linux 7c942938bb42 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 71c21f0
Default Java 1.8.0_212
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/1/testReport/
Max. process+thread count 5386 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 37 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 10 Maven dependency ordering for branch
+1 mvninstall 466 trunk passed
+1 compile 247 trunk passed
+1 checkstyle 59 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 784 branch has no errors when building and testing our client artifacts.
+1 javadoc 164 trunk passed
0 spotbugs 314 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 508 trunk passed
_ Patch Compile Tests _
0 mvndep 28 Maven dependency ordering for patch
+1 mvninstall 450 the patch passed
+1 compile 278 the patch passed
+1 cc 278 the patch passed
+1 javac 278 the patch passed
+1 checkstyle 85 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 683 patch has no errors when building and testing our client artifacts.
+1 javadoc 164 the patch passed
+1 findbugs 525 the patch passed
_ Other Tests _
-1 unit 194 hadoop-hdds in the patch failed.
-1 unit 1777 hadoop-ozone in the patch failed.
+1 asflicense 53 The patch does not generate ASF License warnings.
6713
Reason Tests
Failed junit tests hadoop.ozone.container.ozoneimpl.TestOzoneContainer
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestWatchForCommit
Subsystem Report/Notes
Docker Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/2/artifact/out/Dockerfile
GITHUB PR #1108
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 20a26e73c615 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 71c21f0
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/2/testReport/
Max. process+thread count 5386 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bharatviswa504 bharatviswa504 self-assigned this Jul 17, 2019
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 39 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 11 new or modified test files.
_ trunk Compile Tests _
0 mvndep 16 Maven dependency ordering for branch
+1 mvninstall 483 trunk passed
+1 compile 269 trunk passed
+1 checkstyle 79 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 863 branch has no errors when building and testing our client artifacts.
+1 javadoc 166 trunk passed
0 spotbugs 321 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 503 trunk passed
_ Patch Compile Tests _
0 mvndep 26 Maven dependency ordering for patch
+1 mvninstall 451 the patch passed
+1 compile 269 the patch passed
+1 cc 269 the patch passed
+1 javac 269 the patch passed
+1 checkstyle 87 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 692 patch has no errors when building and testing our client artifacts.
+1 javadoc 166 the patch passed
+1 findbugs 527 the patch passed
_ Other Tests _
+1 unit 294 hadoop-hdds in the patch passed.
-1 unit 1616 hadoop-ozone in the patch failed.
+1 asflicense 56 The patch does not generate ASF License warnings.
6791
Reason Tests
Failed junit tests hadoop.ozone.container.server.TestSecureContainerServer
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
Subsystem Report/Notes
Docker Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/3/artifact/out/Dockerfile
GITHUB PR #1108
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux cb07dd5ad9f5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d5ef38b
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/3/testReport/
Max. process+thread count 5157 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 36 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 11 new or modified test files.
_ trunk Compile Tests _
0 mvndep 27 Maven dependency ordering for branch
+1 mvninstall 462 trunk passed
+1 compile 254 trunk passed
+1 checkstyle 69 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 785 branch has no errors when building and testing our client artifacts.
+1 javadoc 160 trunk passed
0 spotbugs 312 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 500 trunk passed
_ Patch Compile Tests _
0 mvndep 26 Maven dependency ordering for patch
+1 mvninstall 434 the patch passed
+1 compile 268 the patch passed
+1 cc 268 the patch passed
+1 javac 268 the patch passed
+1 checkstyle 66 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 614 patch has no errors when building and testing our client artifacts.
+1 javadoc 146 the patch passed
+1 findbugs 521 the patch passed
_ Other Tests _
-1 unit 276 hadoop-hdds in the patch failed.
-1 unit 1459 hadoop-ozone in the patch failed.
+1 asflicense 38 The patch does not generate ASF License warnings.
6294
Reason Tests
Failed junit tests hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.container.server.TestSecureContainerServer
Subsystem Report/Notes
Docker Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/4/artifact/out/Dockerfile
GITHUB PR #1108
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 60075cfa943e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 4e66cb9
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/4/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/4/testReport/
Max. process+thread count 5283 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/4/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7
arp7 approved these changes Jul 19, 2019
View reviewed changes
Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor question. +1 otherwise.

@bharatviswa504
Copy link
Contributor Author

bharatviswa504 commented Jul 19, 2019 •
edited
Loading

Thank You @arp7 for the review.
Test failures are not related to this patch.
I will commit this shortly.

@bharatviswa504
Copy link
Contributor Author

Rebased it with the latest trunk.
Will commit this after CI.

@bharatviswa504
Copy link
Contributor Author

/retest

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 36 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 1 The patch appears to include 7 new or modified test files.
_ trunk Compile Tests _
0 mvndep 26 Maven dependency ordering for branch
+1 mvninstall 466 trunk passed
+1 compile 250 trunk passed
+1 checkstyle 66 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 840 branch has no errors when building and testing our client artifacts.
+1 javadoc 157 trunk passed
0 spotbugs 332 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 531 trunk passed
_ Patch Compile Tests _
0 mvndep 22 Maven dependency ordering for patch
+1 mvninstall 459 the patch passed
+1 compile 268 the patch passed
+1 cc 268 the patch passed
+1 javac 268 the patch passed
+1 checkstyle 69 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 633 patch has no errors when building and testing our client artifacts.
+1 javadoc 167 the patch passed
+1 findbugs 596 the patch passed
_ Other Tests _
+1 unit 315 hadoop-hdds in the patch passed.
-1 unit 2689 hadoop-ozone in the patch failed.
+1 asflicense 46 The patch does not generate ASF License warnings.
7795
Reason Tests
Failed junit tests hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestReadRetries
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.container.server.TestSecureContainerServer
Subsystem Report/Notes
Docker Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/5/artifact/out/Dockerfile
GITHUB PR #1108
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 43eca5e4b0f8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / f5b2f75
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/5/testReport/
Max. process+thread count 4215 (vs. ulimit of 5500)
modules C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1108/5/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bharatviswa504
Copy link
Contributor Author

Test failures are not related to this patch.
Thank You @arp7 for the review. I will commit this to the trunk.

@bharatviswa504 bharatviswa504 merged commit 4aa76e3 into apache:trunk Jul 20, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
@dnishimura @mynameborat
SAMZA-2277: Semantics for cluster-manager.container.retry.window.ms n…
cf0ecea 
…ot reflected in code

In the current code, the window is only applied and checked on the last retry. However, the check should be done at all retries.

This was found during apache#1104

rmatharu please take a look

Author: Daniel Nishimura <dnishimura@gmail.com>

Reviewers: Ray Matharu <rmatharu@linkedin.com>

Closes apache#1108 from dnishimura/samza-2277-cluster-manager-retry-window-bug-fix and squashes the following commits:

23db835e [Daniel Nishimura] Address minor comments from @rmatharu
0a7c9270 [Daniel Nishimura] Trigger build b/c of flakey test.
0c56499d [Daniel Nishimura] Fix checkstyle
2674f75e [Daniel Nishimura] Address @rmatharu's comments.
ecfe63eb [Daniel Nishimura] SAMZA-2277: Semantics for cluster-manager.container.retry.window.ms not reflected in code
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
@bharatviswa504
HDDS-1805. Implement S3 Initiate MPU request to use Cache and DoubleB…
2643508 
…uffer. (apache#1108)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arp7 arp7 arp7 approved these changes

Assignees

@bharatviswa504 bharatviswa504

Labels
ozone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bharatviswa504 @hadoop-yetus @arp7