HADOOP-16058 S3A to support terasort

[steveloughran]

HADOOP-16058 S3A to support terasort

Change-Id: Icd7911bd44d1382287808f60566a2405265adbd4

[steveloughran]

tested: S3 london, s3guard, versioned bucket. No noted problems

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	30	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 19 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	90	Maven dependency ordering for branch
+1	mvninstall	1435	trunk passed
+1	compile	1221	trunk passed
+1	checkstyle	243	trunk passed
+1	mvnsite	145	trunk passed
+1	shadedclient	1223	branch has no errors when building and testing our client artifacts.
+1	findbugs	155	trunk passed
+1	javadoc	92	trunk passed
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
+1	mvninstall	100	the patch passed
+1	compile	1112	the patch passed
-1	javac	1112	root generated 1 new + 1483 unchanged - 0 fixed = 1484 total (was 1483)
-0	checkstyle	239	root: The patch generated 10 new + 77 unchanged - 0 fixed = 87 total (was 77)
+1	mvnsite	142	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	800	patch has no errors when building and testing our client artifacts.
+1	findbugs	184	the patch passed
+1	javadoc	85	the patch passed
		_ Other Tests _
-1	unit	7519	hadoop-mapreduce-client-jobclient in the patch failed.
+1	unit	56	hadoop-mapreduce-examples in the patch passed.
+1	unit	293	hadoop-aws in the patch passed.
-1	asflicense	60	The patch generated 1 ASF License warnings.
		15149

Reason	Tests
Failed junit tests	hadoop.mapred.TestMRTimelineEventHandling

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/artifact/out/Dockerfile
GITHUB PR	#577
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 7823fae2cae6 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / e026041
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
findbugs	v3.1.0-RC1
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/artifact/out/diff-compile-javac-root.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	1183 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-aws U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-577/1/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

[steveloughran]

javac warning is about a deprected use of makeQualified(FileSystem); no plans to fix.

[ajfabbri]

How do you usually apply these pull requests locally for testing? Anything easier than (1) adding remote (2) checkout branch from new remote, etc.? Almost wish there was a "click to download .patch" here.

[ajfabbri]

+1 LGTM but have not tested yet. Working on that now.

[ajfabbri]

pulled your branch and ran into other compile issues

[ERROR]   TestZKSignerSecretProvider.testMultiple1:247->testMultiple:301 NoSuchMethod or...
[ERROR]   TestZKSignerSecretProvider.testMultiple2:252->testMultiple:301 NoSuchMethod or...
[ERROR]   TestZKSignerSecretProvider.testOne:87 NoSuchMethod org.mockito.Mockito.timeout...
[INFO]
[ERROR] Tests run: 138, Failures: 0, Errors: 3, Skipped: 0```

[steveloughran]

thanks for the review. I saw that test on yetus on the hadoop JIRA patch submission. I'm not going anywhere near that module or any of the POM changs, so I can't come up with any theory why they will have broken. Maybe your .m2 cache has got into a mess?

[steveloughran]

given I've got that +1, is it OK for me to commit if I do a local retest? Just trying to keep my personal-PR-count under control.

[steveloughran]

@ajfabbri

• you know for any pr, put .patch at the end and you get the patch
• if you hit the "view command line instructions" you get the shell commands to checkout the branch. Probably best strategy for an evolving pr
• And you can fetch the PR from the project by ID: git fetch origin pull/577/head:BRANCHNAME

[steveloughran]

on trunk, the yarn smart-apply-patch works for me, provided I don't ask it to actually commit the PR

dev-support/bin/smart-apply-patch --project=hadoop GH:577

This will pull down the patch, apply it. doesn't seem to to work for older branches though...you'd probably need to update it everywhere to the latest yetus code.

[ajfabbri]

Yep +1 for you to commit it with a local re-test. Thanks for the contribution. Also thanks for the smart-apply-patch tip!

[steveloughran]

VS.code seems to have a github plugin for in-IDE reviewing: need to play with that. If you really can review & comments from in an IDE, that'd be wonderful