MAPREDUCE-7237. Supports config the shuffle's path cache related parameters

[jiwq]

Nowadays the ShuffleHandler#Shuffle#pathCache's related parameters is hard coding. We should support config these.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	46	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1253	trunk passed
+1	compile	21	trunk passed
+1	checkstyle	19	trunk passed
+1	mvnsite	23	trunk passed
+1	shadedclient	763	branch has no errors when building and testing our client artifacts.
+1	javadoc	20	trunk passed
0	spotbugs	37	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	33	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	21	the patch passed
+1	compile	16	the patch passed
+1	javac	16	the patch passed
-0	checkstyle	12	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle: The patch generated 1 new + 60 unchanged - 5 fixed = 61 total (was 65)
+1	mvnsite	18	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	829	patch has no errors when building and testing our client artifacts.
+1	javadoc	18	the patch passed
+1	findbugs	42	the patch passed
		_ Other Tests _
+1	unit	27	hadoop-mapreduce-client-shuffle in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3306

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1397/1/artifact/out/Dockerfile
GITHUB PR	#1397
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 048ac159b3dc 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / dc9abd2
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1397/1/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1397/1/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1397/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[aajisaka]

Hi @jiwq, I have two questions:

• Have you tuned the parameters in your environment?
• If the answer is yes, is there any positive effect?

As Jian He commented in https://issues.apache.org/jira/browse/MAPREDUCE-6197?focusedCommentId=15333791&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15333791

If we found any issue with current settings/policy in large production deployment in future, we can change both side together.

I'd like to know it's the time to make the paramters configurable or not.

[aajisaka]

In addition, would you document the new parameters in mapred-default.xml?

[jiwq]

Thanks @aajisaka for review.

Have you tuned the parameters in your environment? If the answer is yes, is there any positive effect?

Yes, it can reduce disk IO. Also, in my opinion it should better can be configured by user, because the default params cannot adjust all situations.

In addition, would you document the new parameters in mapred-default.xml?

Added.

[aajisaka]

+1