HADOOP-16384: Avoid inconsistencies between DDB and S3 #1003

steveloughran · 2019-06-21T12:37:09Z

Step one: a new root test to execute prune against the live store and DDB table.

To follow: debug the conditions leading to prune fail on my store :)

Change-Id: I25d138ed7b42f2b5eda12c2cd736f8c054b69820

hadoop-yetus · 2019-06-21T13:31:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	29	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1009	trunk passed
+1	compile	35	trunk passed
+1	checkstyle	22	trunk passed
+1	mvnsite	35	trunk passed
+1	shadedclient	679	branch has no errors when building and testing our client artifacts.
+1	javadoc	26	trunk passed
0	spotbugs	53	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	52	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	31	the patch passed
+1	compile	29	the patch passed
+1	javac	29	the patch passed
-0	checkstyle	18	hadoop-tools/hadoop-aws: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3)
+1	mvnsite	33	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	705	patch has no errors when building and testing our client artifacts.
+1	javadoc	20	the patch passed
+1	findbugs	60	the patch passed
		_ Other Tests _
+1	unit	273	hadoop-aws in the patch passed.
+1	asflicense	26	The patch does not generate ASF License warnings.
		3185

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/1/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux ecca0186b66c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `256bdd6`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/1/testReport/
Max. process+thread count	447 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

hadoop-yetus · 2019-06-25T23:48:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	32	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	27	Maven dependency ordering for branch
+1	mvninstall	1158	trunk passed
+1	compile	1051	trunk passed
+1	checkstyle	144	trunk passed
+1	mvnsite	132	trunk passed
+1	shadedclient	1023	branch has no errors when building and testing our client artifacts.
+1	javadoc	99	trunk passed
0	spotbugs	68	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	184	trunk passed
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	988	the patch passed
+1	javac	988	the patch passed
-0	checkstyle	145	root: The patch generated 8 new + 5 unchanged - 0 fixed = 13 total (was 5)
+1	mvnsite	130	the patch passed
-1	whitespace	0	The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	687	patch has no errors when building and testing our client artifacts.
+1	javadoc	102	the patch passed
-1	findbugs	77	hadoop-tools/hadoop-aws generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
		_ Other Tests _
+1	unit	527	hadoop-common in the patch passed.
+1	unit	288	hadoop-aws in the patch passed.
+1	asflicense	53	The patch does not generate ASF License warnings.
		7094

Reason	Tests
FindBugs	module:hadoop-tools/hadoop-aws
	Found reliance on default encoding in org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.execute():in org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.execute(): new java.io.PrintWriter(File) At DumpS3GuardTable.java:[line 127]
	Exception is caught when Exception is not thrown in org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.execute() At DumpS3GuardTable.java:is not thrown in org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.execute() At DumpS3GuardTable.java:[line 141]
	Found reliance on default encoding in new org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable$CsvFile(FileSystem, Path, boolean, String, String, String):in new org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable$CsvFile(FileSystem, Path, boolean, String, String, String): new java.io.PrintWriter(OutputStream) At DumpS3GuardTable.java:[line 318]

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/2/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux e2a4dc9d5e4a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `4b50981`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/2/artifact/out/diff-checkstyle-root.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/2/artifact/out/whitespace-eol.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/2/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/2/testReport/
Max. process+thread count	1447 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

hadoop-yetus · 2019-06-26T19:09:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	33	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	30	Maven dependency ordering for branch
+1	mvninstall	1119	trunk passed
+1	compile	1141	trunk passed
+1	checkstyle	146	trunk passed
+1	mvnsite	134	trunk passed
+1	shadedclient	1035	branch has no errors when building and testing our client artifacts.
+1	javadoc	103	trunk passed
0	spotbugs	69	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	190	trunk passed
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	986	the patch passed
+1	javac	986	the patch passed
-0	checkstyle	140	root: The patch generated 4 new + 38 unchanged - 1 fixed = 42 total (was 39)
+1	mvnsite	126	the patch passed
-1	whitespace	0	The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	595	patch has no errors when building and testing our client artifacts.
+1	javadoc	87	the patch passed
+1	findbugs	190	the patch passed
		_ Other Tests _
+1	unit	531	hadoop-common in the patch passed.
+1	unit	280	hadoop-aws in the patch passed.
+1	asflicense	37	The patch does not generate ASF License warnings.
		7007

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/3/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 9f3d876ba7ba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `062eb60`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/3/artifact/out/diff-checkstyle-root.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/3/artifact/out/whitespace-eol.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/3/testReport/
Max. process+thread count	1447 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

hadoop-yetus · 2019-06-26T19:13:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	34	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	71	Maven dependency ordering for branch
+1	mvninstall	1078	trunk passed
+1	compile	1052	trunk passed
+1	checkstyle	144	trunk passed
+1	mvnsite	135	trunk passed
+1	shadedclient	1021	branch has no errors when building and testing our client artifacts.
+1	javadoc	108	trunk passed
0	spotbugs	70	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	192	trunk passed
		_ Patch Compile Tests _
0	mvndep	28	Maven dependency ordering for patch
+1	mvninstall	77	the patch passed
+1	compile	988	the patch passed
+1	javac	988	the patch passed
-0	checkstyle	144	root: The patch generated 4 new + 38 unchanged - 1 fixed = 42 total (was 39)
+1	mvnsite	130	the patch passed
-1	whitespace	0	The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	690	patch has no errors when building and testing our client artifacts.
+1	javadoc	109	the patch passed
+1	findbugs	206	the patch passed
		_ Other Tests _
-1	unit	509	hadoop-common in the patch failed.
+1	unit	298	hadoop-aws in the patch passed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		7094

Reason	Tests
Failed junit tests	hadoop.util.TestDiskChecker

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/4/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 022e45aee381 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `062eb60`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/4/artifact/out/diff-checkstyle-root.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/4/artifact/out/whitespace-eol.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/4/testReport/
Max. process+thread count	1347 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

hadoop-yetus · 2019-07-01T14:32:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	32	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	65	Maven dependency ordering for branch
+1	mvninstall	1054	trunk passed
+1	compile	1036	trunk passed
+1	checkstyle	140	trunk passed
+1	mvnsite	129	trunk passed
+1	shadedclient	989	branch has no errors when building and testing our client artifacts.
+1	javadoc	90	trunk passed
0	spotbugs	81	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	197	trunk passed
		_ Patch Compile Tests _
0	mvndep	30	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	1001	the patch passed
+1	javac	1001	the patch passed
-0	checkstyle	132	root: The patch generated 5 new + 39 unchanged - 1 fixed = 44 total (was 40)
+1	mvnsite	116	the patch passed
-1	whitespace	0	The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	647	patch has no errors when building and testing our client artifacts.
+1	javadoc	96	the patch passed
+1	findbugs	199	the patch passed
		_ Other Tests _
+1	unit	536	hadoop-common in the patch passed.
+1	unit	295	hadoop-aws in the patch passed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		6940

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/6/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 6f8df5daac40 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1e727cf`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/6/artifact/out/diff-checkstyle-root.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/6/artifact/out/whitespace-eol.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/6/testReport/
Max. process+thread count	1422 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-01T14:45:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	110	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	72	Maven dependency ordering for branch
+1	mvninstall	1370	trunk passed
+1	compile	1327	trunk passed
+1	checkstyle	196	trunk passed
+1	mvnsite	158	trunk passed
+1	shadedclient	1263	branch has no errors when building and testing our client artifacts.
+1	javadoc	131	trunk passed
0	spotbugs	89	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	241	trunk passed
		_ Patch Compile Tests _
0	mvndep	28	Maven dependency ordering for patch
+1	mvninstall	107	the patch passed
+1	compile	1195	the patch passed
+1	javac	1195	the patch passed
-0	checkstyle	153	root: The patch generated 5 new + 39 unchanged - 1 fixed = 44 total (was 40)
+1	mvnsite	124	the patch passed
-1	whitespace	0	The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	752	patch has no errors when building and testing our client artifacts.
+1	javadoc	97	the patch passed
+1	findbugs	203	the patch passed
		_ Other Tests _
-1	unit	519	hadoop-common in the patch failed.
+1	unit	295	hadoop-aws in the patch passed.
+1	asflicense	45	The patch does not generate ASF License warnings.
		8354

Reason	Tests
Failed junit tests	hadoop.security.TestFixKerberosTicketOrder

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/5/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 804c87ed4b49 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1e727cf`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/5/artifact/out/diff-checkstyle-root.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/5/artifact/out/whitespace-eol.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/5/testReport/
Max. process+thread count	1343 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

hadoop-yetus · 2019-07-02T15:07:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	31	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 8 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	66	Maven dependency ordering for branch
+1	mvninstall	1058	trunk passed
+1	compile	1062	trunk passed
+1	checkstyle	136	trunk passed
+1	mvnsite	121	trunk passed
+1	shadedclient	929	branch has no errors when building and testing our client artifacts.
+1	javadoc	90	trunk passed
0	spotbugs	63	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	177	trunk passed
		_ Patch Compile Tests _
0	mvndep	23	Maven dependency ordering for patch
+1	mvninstall	74	the patch passed
+1	compile	1019	the patch passed
+1	javac	1019	the patch passed
-0	checkstyle	151	root: The patch generated 10 new + 56 unchanged - 2 fixed = 66 total (was 58)
+1	mvnsite	117	the patch passed
-1	whitespace	0	The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	647	patch has no errors when building and testing our client artifacts.
-1	javadoc	28	hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
-1	findbugs	82	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
-1	unit	571	hadoop-common in the patch failed.
+1	unit	293	hadoop-aws in the patch passed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		6913

Reason	Tests
FindBugs	module:hadoop-tools/hadoop-aws
	Private method org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.fail(String, Throwable) is never called At DumpS3GuardTable.java:never called At DumpS3GuardTable.java:[line 416]
Failed junit tests	hadoop.ha.TestZKFailoverController

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux ac1a89cc3a39 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / e966edd
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/artifact/out/diff-checkstyle-root.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/artifact/out/whitespace-eol.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/testReport/
Max. process+thread count	1422 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/8/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…ystem in a DDB table * Needs a -force argument, otherwise it's just a dry run * With tests * And coverage in the testing docs Change-Id: I9b0e5edf8dd728edc2b007535140bc09a566d9a4

Change-Id: Idb3c7406ede62a02123a7d4ea66e77dd107d4f47

…eprecated exception Change-Id: Ifc7babd679d157e6db20aa9a0c78b22188100fae

DDB store now has a special log s3guard.Operations which logs PUT/DELETE/TOMBSTONE calls with the application state; that state now has a counter to help provide context DumpS3GuardTable handles situation where a file isn't there when you look closer Change-Id: I49e0e65d6a65adbdf4ec048484bf323a9332d89e

This code is changed based on the hypothesis that it's been the pruning which has been rendering the store inconsistent in parallel test runs. Change-Id: Iab33d7eb57891f6ab92a12e4c75db12bfd794d75

…le rename speedups * The final cause of tombstones seems to be in ITestS3GuardListConsistency whose test files were appearing the most. Changes: * turn the FS into consistent mode before test teardown * explicitly delete the individually created files, so that in the rename() test we were guaranteeing that they were being deleted Made the turning back on of the rename speedup features a compile time option; turned back on, tests are happy. Planning more retests of this to see that all has gone quiet. Change-Id: If341687a1cbdc905782862f36b473faa9a0df84c

* reinstate all the disabled HADOOP-15183 ancestor caching * Add a test to explicitly verify that renaming an empty dir removes the marker from S3 * compare with trunk to reduce difference * Add javadocs on new dump and purge classes Change-Id: I80abc8e022a92bbb163226aa4dfa6ef80fcc2b0e

Change-Id: I13734049ddabdf3a451c906ca3fd2780d4b49c51

* summary prints the results * use queues to do the treewalks Change-Id: I4544ace79a5f004141bd81f0fb7a1849fa4b89c0 ITestS3GuardListConsistency: isolates an intermittently failing test within its own subtree ITestS3GuardRootOperations: turn on the pruning

* AbstractITCommitMRJob disables FS caching * ITestS3GuardDDBRootOperations disables some superfluous calls which added ~30s to the runtime Change-Id: Ic5b5f7bdd509f0fbaeb4b7d578a3060515e3b963

…stic Change-Id: I3a7bd5a76d1bb24d1a8d66a694310f065b5826c4

With the S3Guardroot operations test, there's no need to have a test of the Dump operation in ITestS3GuardToolDynamoDB; all it does is trigger test timeouts over slow network connections. Prune tool usage entry had "tombstone" entry in wrong place and without the leading "-" Change-Id: Ib71fcf077d407bc1ea5b333044fbb68c86642921

…tion (for yetus) Change-Id: I3652f2da54e6543dd7e2fa556d50b6b570db5b42

steveloughran · 2019-07-11T17:28:32Z

Update:

Rebased on trunk, which, with HADOOP-16357, is more informative on terasort test failures.
removed that check for class instance of in the ITestDynamoDBMetadataStoreScale

hadoop-yetus · 2019-07-11T18:44:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	47	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 16 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	32	Maven dependency ordering for branch
+1	mvninstall	1059	trunk passed
+1	compile	1034	trunk passed
+1	checkstyle	143	trunk passed
+1	mvnsite	127	trunk passed
+1	shadedclient	938	branch has no errors when building and testing our client artifacts.
+1	javadoc	87	trunk passed
0	spotbugs	66	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	176	trunk passed
-0	patch	97	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
+1	mvninstall	79	the patch passed
+1	compile	1063	the patch passed
+1	javac	1063	root generated 0 new + 1475 unchanged - 2 fixed = 1475 total (was 1477)
-0	checkstyle	144	root: The patch generated 9 new + 66 unchanged - 2 fixed = 75 total (was 68)
+1	mvnsite	118	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	625	patch has no errors when building and testing our client artifacts.
+1	javadoc	86	the patch passed
+1	findbugs	203	the patch passed
		_ Other Tests _
+1	unit	560	hadoop-common in the patch passed.
-1	unit	289	hadoop-aws in the patch failed.
+1	asflicense	40	The patch does not generate ASF License warnings.
		6873

Reason	Tests
Failed junit tests	hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/17/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux b805183e697e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5747f6c`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/17/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/17/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/17/testReport/
Max. process+thread count	1403 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/17/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-07-11T19:01:48Z

checkstyles are invalid or, in the case of an intentation one, incorrect.

steveloughran · 2019-07-11T19:04:24Z

Unit test failure looks like there was a mismatch between hadoop-common and hadoop-aws builds, as this is the test which HADOOP-16357 changed. The line at fault != the current source either.

[ERROR] Failures: 
[ERROR]   TestStagingPartitionedTaskCommit.testDefault:91->Assert.assertEquals:118->Assert.failNotEquals:834->Assert.fail:88 Conflict resolution mode in PartitionedStagingCommitter{StagingCommitter{AbstractS3ACommitter{role=Task committer attempt_job_0001_r_000002_3, name=partitioned, outputPath=s3a://bucket-name/output/path, workPath=file:/tmp/buffer/9d9bc724-9b38-4e8d-9e31-091425da9872/_temporary/0/_temporary/attempt_job_0001_r_000002_3}, conflictResolution=APPEND, wrappedCommitter=FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_job_0001}; taskId=attempt_job_0001_r_000002_3, status=''}; org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter@5684b9d4}; outputPath=file:/tmp/cluster/jenkins/9d9bc724-9b38-4e8d-9e31-091425da9872/staging-uploads, workPath=null, algorithmVersion=1, skipCleanup=false, ignoreCleanupFailures=false}}} expected:<FAIL> but was:<APPEND>

steveloughran · 2019-07-11T19:05:29Z

Tests run locally, Dparallel-tests -DtestsThreadCount=12 -Ds3guard -Ddynamo -Dauth all tests succeeded. This is as good as it gets. Rerunning in non-auth

bgaborg · 2019-07-11T19:17:11Z

I'll test this against Ireland and report back with the results. Thanks for working on this @steveloughran !

hadoop-yetus · 2019-07-11T19:23:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	39	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	1	The patch appears to include 16 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	23	Maven dependency ordering for branch
+1	mvninstall	1038	trunk passed
+1	compile	1110	trunk passed
+1	checkstyle	136	trunk passed
+1	mvnsite	117	trunk passed
+1	shadedclient	920	branch has no errors when building and testing our client artifacts.
+1	javadoc	89	trunk passed
0	spotbugs	60	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	170	trunk passed
-0	patch	96	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
+1	mvninstall	78	the patch passed
+1	compile	1022	the patch passed
+1	javac	1022	root generated 0 new + 1474 unchanged - 2 fixed = 1474 total (was 1476)
-0	checkstyle	148	root: The patch generated 8 new + 66 unchanged - 2 fixed = 74 total (was 68)
+1	mvnsite	130	the patch passed
+1	whitespace	1	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	700	patch has no errors when building and testing our client artifacts.
+1	javadoc	108	the patch passed
+1	findbugs	208	the patch passed
		_ Other Tests _
+1	unit	524	hadoop-common in the patch passed.
+1	unit	287	hadoop-aws in the patch passed.
+1	asflicense	53	The patch does not generate ASF License warnings.
		6971

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/18/artifact/out/Dockerfile
GITHUB PR	#1003
JIRA Issue	HADOOP-16384
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux cff2d2145215 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6a3433b`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/18/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/18/testReport/
Max. process+thread count	1446 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/18/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bgaborg · 2019-07-11T21:20:19Z

tested against ireland:

[ERROR] Errors:
[ERROR]   ITestMagicCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound
[ERROR]   ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound
[ERROR]   ITestPartitionCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound
[ERROR]   ITestStagingCommitMRJob>AbstractITCommitMRJob.testMRJob:146->AbstractFSContractTestBase.assertIsDirectory:327 ? FileNotFound

otherwise one timeout not related to this.

bgaborg · 2019-07-11T21:21:03Z

I created a jira for the testMRJob failures as I'm the one who gets it - I should fix it.

steveloughran · 2019-07-12T11:42:46Z

thanks. Gabor, you do explicitly need to do that +1 for the record

bgaborg · 2019-07-12T11:44:35Z

+1; these improvements will make the s3a more stable.

steveloughran · 2019-07-12T12:06:56Z

thanks for all the testing -committed to trunk

…ed on if log4j or log4j2 is class-loaded Removing job.diagnostics.appender.class config, and populating it based on if log4j or log4j2 is class-loaded, defaults to assuming log4j. Author: Ray Matharu <rmatharu@linkedin.com> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#1003 from rmatharu/simplifying-diagnostics-config

steveloughran added fs/s3 changes related to hadoop-aws; submitter must declare test endpoint work-in-progress labels Jun 21, 2019

hadoop-yetus reviewed Jun 25, 2019

View reviewed changes