HDDS-1942. Support copy during S3 multipart upload part creation #1279

elek · 2019-08-11T12:45:10Z

Uploads a part by copying data from an existing object as data source

Documented here:

https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html

See: https://issues.apache.org/jira/browse/HDDS-1942

hadoop-yetus · 2019-08-11T15:19:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	65	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	31	Maven dependency ordering for branch
+1	mvninstall	666	trunk passed
+1	compile	412	trunk passed
+1	checkstyle	73	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	970	branch has no errors when building and testing our client artifacts.
+1	javadoc	172	trunk passed
0	spotbugs	477	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	698	trunk passed
		_ Patch Compile Tests _
0	mvndep	19	Maven dependency ordering for patch
+1	mvninstall	584	the patch passed
+1	compile	381	the patch passed
+1	javac	381	the patch passed
+1	checkstyle	71	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	671	patch has no errors when building and testing our client artifacts.
+1	javadoc	156	the patch passed
+1	findbugs	704	the patch passed
		_ Other Tests _
+1	unit	334	hadoop-hdds in the patch passed.
-1	unit	2991	hadoop-ozone in the patch failed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		9182

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
	hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
	hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
	hadoop.ozone.client.rpc.Test2WayCommitInRatis
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/1/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux a8dd26e8e4a8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `cf5d895`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/1/testReport/
Max. process+thread count	5065 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

lokeshj1703

@elek Thanks for working on this! The changes look good to me.
The acceptance test failures seem related. Can we also add a unit test related to same?

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot

bharatviswa504

And also there are few parameters related to x-amz-copy-source, these are not implemented.

Related to eTag I think we don't have backend implemented but for x-amz-copy-source-if-unmodified-since we can use lastModified from KeyInfo and do this right?

hadoop-yetus · 2019-08-16T15:44:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	26	Maven dependency ordering for branch
+1	mvninstall	576	trunk passed
+1	compile	348	trunk passed
+1	checkstyle	76	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	840	branch has no errors when building and testing our client artifacts.
+1	javadoc	154	trunk passed
0	spotbugs	452	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	664	trunk passed
		_ Patch Compile Tests _
0	mvndep	18	Maven dependency ordering for patch
+1	mvninstall	583	the patch passed
+1	compile	393	the patch passed
+1	javac	393	the patch passed
+1	checkstyle	71	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	670	patch has no errors when building and testing our client artifacts.
+1	javadoc	165	the patch passed
+1	findbugs	685	the patch passed
		_ Other Tests _
+1	unit	287	hadoop-hdds in the patch passed.
-1	unit	1613	hadoop-ozone in the patch failed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		7379

Reason	Tests
Failed junit tests	hadoop.ozone.TestMiniChaosOzoneCluster
	hadoop.ozone.client.rpc.Test2WayCommitInRatis
	hadoop.ozone.client.rpc.TestFailureHandlingByClient
	hadoop.hdds.scm.pipeline.TestNode2PipelineMap

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/2/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 82a979cf92cb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `9b8359b`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/2/testReport/
Max. process+thread count	3935 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-20T14:02:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1147	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	48	Maven dependency ordering for branch
+1	mvninstall	698	trunk passed
+1	compile	419	trunk passed
+1	checkstyle	76	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	884	branch has no errors when building and testing our client artifacts.
+1	javadoc	162	trunk passed
0	spotbugs	481	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	705	trunk passed
		_ Patch Compile Tests _
0	mvndep	19	Maven dependency ordering for patch
+1	mvninstall	586	the patch passed
+1	compile	407	the patch passed
+1	javac	407	the patch passed
+1	checkstyle	87	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	706	patch has no errors when building and testing our client artifacts.
+1	javadoc	164	the patch passed
+1	findbugs	751	the patch passed
		_ Other Tests _
+1	unit	365	hadoop-hdds in the patch passed.
-1	unit	507	hadoop-ozone in the patch failed.
+1	asflicense	38	The patch does not generate ASF License warnings.
		7905

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/3/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 330fe7eaf930 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 6244502
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/3/testReport/
Max. process+thread count	1382 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-08-21T09:15:41Z

Thanks the review @lokeshj1703.

Can we also add a unit test related to same?

Sure, I created a real unit test. Will push it soon...

elek · 2019-08-21T09:28:44Z

Thanks the review @bharatviswa504

Related to eTag I think we don't have backend implemented but for x-amz-copy-source-if-unmodified-since we can use lastModified from KeyInfo and do this right?

Yes, it's true. Let's do it in a separated jira (opened HDDS-1997). Can be better to do it in a separated phase as it's not a blocking feature for using ozone as a docker registry backend, and also it requires more code (especially more testing code) and can be easier to review in a separated step.

elek · 2019-08-21T09:58:40Z

Thanks the review @bharatviswa504

Related to eTag I think we don't have backend implemented but for x-amz-copy-source-if-unmodified-since we can use lastModified from KeyInfo and do this right?

Yes, it's true. Let's do it in a separated jira (opened HDDS-1997). Can be better to do it in a separated phase as it's not a blocking feature for using ozone as a docker registry backend, and also it requires more code (especially more testing code) and can be easier to review in a separated step.

hadoop-yetus · 2019-08-21T13:13:25Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	84	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	11	Maven dependency ordering for branch
+1	mvninstall	631	trunk passed
+1	compile	370	trunk passed
+1	checkstyle	61	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	826	branch has no errors when building and testing our client artifacts.
+1	javadoc	155	trunk passed
0	spotbugs	424	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	612	trunk passed
		_ Patch Compile Tests _
0	mvndep	19	Maven dependency ordering for patch
+1	mvninstall	554	the patch passed
+1	compile	361	the patch passed
+1	javac	361	the patch passed
-0	checkstyle	34	hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	677	patch has no errors when building and testing our client artifacts.
+1	javadoc	150	the patch passed
+1	findbugs	631	the patch passed
		_ Other Tests _
+1	unit	352	hadoop-hdds in the patch passed.
-1	unit	3260	hadoop-ozone in the patch failed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		9038

Reason	Tests
Failed junit tests	hadoop.ozone.container.server.TestSecureContainerServer
	hadoop.ozone.scm.TestXceiverClientManager
	hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
	hadoop.ozone.TestMiniOzoneCluster
	hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestReadRetries

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/4/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 31f3792bf7c2 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `10ec31d`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/4/artifact/out/diff-checkstyle-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/4/testReport/
Max. process+thread count	4199 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-22T13:16:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	26	Maven dependency ordering for branch
+1	mvninstall	598	trunk passed
+1	compile	340	trunk passed
+1	checkstyle	62	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	820	branch has no errors when building and testing our client artifacts.
+1	javadoc	163	trunk passed
0	spotbugs	432	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	630	trunk passed
		_ Patch Compile Tests _
0	mvndep	18	Maven dependency ordering for patch
+1	mvninstall	574	the patch passed
+1	compile	385	the patch passed
+1	javac	385	the patch passed
-0	checkstyle	36	hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	649	patch has no errors when building and testing our client artifacts.
+1	javadoc	154	the patch passed
+1	findbugs	674	the patch passed
		_ Other Tests _
+1	unit	293	hadoop-hdds in the patch passed.
-1	unit	2091	hadoop-ozone in the patch failed.
+1	asflicense	39	The patch does not generate ASF License warnings.
		7768

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
	hadoop.ozone.container.server.TestSecureContainerServer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/5/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6745475e8bcc 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ee7c261`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/5/artifact/out/diff-checkstyle-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/5/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/5/testReport/
Max. process+thread count	5301 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bharatviswa504 · 2019-08-23T06:27:02Z

+1 LGTM.
Need to verify test failures, for acceptance test not able to verify them, as it is opening an html code contained in Github.

hadoop-yetus · 2019-08-27T09:42:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	17	Maven dependency ordering for branch
+1	mvninstall	622	trunk passed
+1	compile	382	trunk passed
+1	checkstyle	83	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	848	branch has no errors when building and testing our client artifacts.
+1	javadoc	179	trunk passed
0	spotbugs	454	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	660	trunk passed
		_ Patch Compile Tests _
0	mvndep	21	Maven dependency ordering for patch
-1	mvninstall	288	hadoop-ozone in the patch failed.
-1	compile	80	hadoop-ozone in the patch failed.
-1	javac	80	hadoop-ozone in the patch failed.
-0	checkstyle	42	hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	659	patch has no errors when building and testing our client artifacts.
+1	javadoc	169	the patch passed
-1	findbugs	164	hadoop-ozone in the patch failed.
		_ Other Tests _
+1	unit	332	hadoop-hdds in the patch passed.
-1	unit	349	hadoop-ozone in the patch failed.
+1	asflicense	40	The patch does not generate ASF License warnings.
		5629

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b468e8002e70 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3329257`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/patch-compile-hadoop-ozone.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/diff-checkstyle-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/testReport/
Max. process+thread count	1242 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-28T07:23:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	193	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	23	Maven dependency ordering for branch
+1	mvninstall	839	trunk passed
+1	compile	467	trunk passed
+1	checkstyle	112	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1192	branch has no errors when building and testing our client artifacts.
+1	javadoc	233	trunk passed
0	spotbugs	590	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	875	trunk passed
		_ Patch Compile Tests _
0	mvndep	26	Maven dependency ordering for patch
+1	mvninstall	731	the patch passed
+1	compile	471	the patch passed
+1	javac	471	the patch passed
+1	checkstyle	91	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	874	patch has no errors when building and testing our client artifacts.
+1	javadoc	216	the patch passed
+1	findbugs	919	the patch passed
		_ Other Tests _
+1	unit	445	hadoop-hdds in the patch passed.
-1	unit	2752	hadoop-ozone in the patch failed.
0	asflicense	53	ASF License check generated no output?
		10710

Reason	Tests
Failed junit tests	hadoop.ozone.om.TestContainerReportWithKeys
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog
	hadoop.ozone.scm.TestSCMNodeManagerMXBean
	hadoop.hdds.scm.pipeline.TestSCMPipelineManager
	hadoop.ozone.client.rpc.TestKeyInputStream
	hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode
	hadoop.ozone.client.rpc.Test2WayCommitInRatis
	hadoop.ozone.client.rpc.TestCommitWatcher
	hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/7/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c26573fd2250 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b1eee8b`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/7/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/7/testReport/
Max. process+thread count	3973 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/7/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-08-28T14:51:28Z

/retest

elek · 2019-08-28T15:06:44Z

Thanks the review @bharatviswa504

@lokeshj1703 Do you have any more concerns? Your review is still in "requested change" phase but I think I addressed the comments.

elek · 2019-08-28T15:44:24Z

@lokeshj1703 Do you have any more concerns? If not, can you please change the status from "requested change"?

lokeshj1703

@elek Thanks for updating the PR! The changes look good to me. +1

bharatviswa504 · 2019-08-28T18:23:11Z

@elek There is an acceptance test failure in Multipart upload. Not sure if it is related here, as I see that test is passing in CI run.

anuengineer · 2019-08-28T19:03:52Z

Thank you for the reviews. I have committed this to the trunk. @elek Thanks for the contribution.

hadoop-yetus · 2019-08-28T19:53:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	56	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	23	Maven dependency ordering for branch
+1	mvninstall	720	trunk passed
+1	compile	440	trunk passed
+1	checkstyle	84	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1048	branch has no errors when building and testing our client artifacts.
+1	javadoc	170	trunk passed
0	spotbugs	515	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	748	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
+1	mvninstall	638	the patch passed
+1	compile	466	the patch passed
+1	javac	466	the patch passed
+1	checkstyle	87	the patch passed
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	825	patch has no errors when building and testing our client artifacts.
+1	javadoc	189	the patch passed
+1	findbugs	670	the patch passed
		_ Other Tests _
-1	unit	347	hadoop-hdds in the patch failed.
-1	unit	365	hadoop-ozone in the patch failed.
+1	asflicense	110	The patch does not generate ASF License warnings.
		7205

Reason	Tests
Failed junit tests	hadoop.hdds.scm.container.TestReplicationManager
	hadoop.ozone.security.TestOzoneDelegationTokenSecretManager

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/8/artifact/out/Dockerfile
GITHUB PR	#1279
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux fbdf9d4346c0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 48cb583
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/8/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/8/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/8/testReport/
Max. process+thread count	1178 (vs. ulimit of 5500)
modules	C: hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1279/8/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek added the ozone label Aug 11, 2019

lokeshj1703 requested changes Aug 13, 2019

View reviewed changes

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java Outdated Show resolved Hide resolved

bharatviswa504 reviewed Aug 15, 2019

View reviewed changes

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java Show resolved Hide resolved

bharatviswa504 reviewed Aug 15, 2019

View reviewed changes

hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot Show resolved Hide resolved

bharatviswa504 reviewed Aug 15, 2019

View reviewed changes

elek added 4 commits August 27, 2019 23:58

HDDS-1942. Support copy during S3 multipart upload part creation

34fe8f6

Add real unit test for multipart upload with copy

253fe8d

fix ObjectEndpoint.parseSourceHeader

ffcc618

add smoketest for multipart upload with copy + range

9b7b962

elek force-pushed the HDDS-1942 branch from bfff73a to 591346f Compare August 27, 2019 22:14

fix rebase problems

591346f

fix acceptance tests and rat issue

8337d0e

lokeshj1703 approved these changes Aug 28, 2019

View reviewed changes

anuengineer closed this Aug 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-1942. Support copy during S3 multipart upload part creation #1279

HDDS-1942. Support copy during S3 multipart upload part creation #1279

elek commented Aug 11, 2019

hadoop-yetus commented Aug 11, 2019

lokeshj1703 left a comment

bharatviswa504 left a comment

hadoop-yetus commented Aug 16, 2019

hadoop-yetus commented Aug 20, 2019

elek commented Aug 21, 2019

elek commented Aug 21, 2019

elek commented Aug 21, 2019

hadoop-yetus commented Aug 21, 2019

hadoop-yetus commented Aug 22, 2019

bharatviswa504 commented Aug 23, 2019

hadoop-yetus commented Aug 27, 2019

hadoop-yetus commented Aug 28, 2019

elek commented Aug 28, 2019

elek commented Aug 28, 2019

elek commented Aug 28, 2019

lokeshj1703 left a comment

bharatviswa504 commented Aug 28, 2019

anuengineer commented Aug 28, 2019

hadoop-yetus commented Aug 28, 2019

HDDS-1942. Support copy during S3 multipart upload part creation #1279

HDDS-1942. Support copy during S3 multipart upload part creation #1279

Conversation

elek commented Aug 11, 2019

hadoop-yetus commented Aug 11, 2019

lokeshj1703 left a comment

Choose a reason for hiding this comment

bharatviswa504 left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Aug 16, 2019

hadoop-yetus commented Aug 20, 2019

elek commented Aug 21, 2019

elek commented Aug 21, 2019

elek commented Aug 21, 2019

hadoop-yetus commented Aug 21, 2019

hadoop-yetus commented Aug 22, 2019

bharatviswa504 commented Aug 23, 2019

hadoop-yetus commented Aug 27, 2019

hadoop-yetus commented Aug 28, 2019

elek commented Aug 28, 2019

elek commented Aug 28, 2019

elek commented Aug 28, 2019

lokeshj1703 left a comment

Choose a reason for hiding this comment

bharatviswa504 commented Aug 28, 2019

anuengineer commented Aug 28, 2019

hadoop-yetus commented Aug 28, 2019