HDFS-15202 Boost short circuit cache (rebase PR-1884) #2016

pustota2009 · 2020-05-13T06:21:26Z

Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

…massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

hadoop-yetus · 2020-05-13T10:20:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	26m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 59s	Maven dependency ordering for branch
+1 💚	mvninstall	23m 36s	trunk passed
+1 💚	compile	3m 57s	trunk passed
+1 💚	checkstyle	1m 4s	trunk passed
+1 💚	mvnsite	2m 11s	trunk passed
+1 💚	shadedclient	19m 13s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 14s	trunk passed
+0 🆗	spotbugs	3m 6s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 30s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 0s	the patch passed
+1 💚	compile	3m 39s	the patch passed
+1 💚	javac	3m 39s	the patch passed
-0 ⚠️	checkstyle	1m 0s	hadoop-hdfs-project: The patch generated 1 new + 342 unchanged - 5 fixed = 343 total (was 347)
+1 💚	mvnsite	1m 56s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 21s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 6s	the patch passed
+1 💚	findbugs	5m 35s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 57s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	111m 13s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s	The patch does not generate ASF License warnings.
		229m 24s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap
	hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/1/artifact/out/Dockerfile
GITHUB PR	#2016
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux dafaeceacd1b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d60496e`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/1/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/1/testReport/
Max. process+thread count	3415 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

fixed codestyle

jojochuang · 2020-05-13T14:45:54Z

This is in continuation of #1884

hadoop-yetus · 2020-05-13T15:49:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 14s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 4 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 47s	Maven dependency ordering for branch
+1 💚	mvninstall	22m 30s	trunk passed
+1 💚	compile	3m 40s	trunk passed
+1 💚	checkstyle	1m 1s	trunk passed
+1 💚	mvnsite	2m 6s	trunk passed
+1 💚	shadedclient	18m 43s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 11s	trunk passed
+0 🆗	spotbugs	3m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 25s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s	the patch passed
+1 💚	compile	3m 32s	the patch passed
+1 💚	javac	3m 32s	the patch passed
+1 💚	checkstyle	0m 56s	hadoop-hdfs-project: The patch generated 0 new + 342 unchanged - 5 fixed = 342 total (was 347)
+1 💚	mvnsite	1m 55s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 34s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 9s	the patch passed
+1 💚	findbugs	5m 31s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 58s	hadoop-hdfs-client in the patch passed.
-1 ❌	unit	107m 4s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s	The patch does not generate ASF License warnings.
		197m 26s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/2/artifact/out/Dockerfile
GITHUB PR	#2016
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 7e5040973a19 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `743c2e9`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/2/testReport/
Max. process+thread count	3185 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jojochuang

+1

jojochuang · 2020-05-15T17:46:34Z

...oject/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/client/impl/TestBlockReaderLocal.java


  public void runBlockReaderLocalTest(BlockReaderLocalTest test,
-      boolean checksum, long readahead) throws IOException {
+      boolean checksum, long readahead, int shortCircuitCachesNum)


A lot of test code changes are not needed if you add an overriding method runBlockReaderLocalTest(BlockReaderLocalTest test, boolean checksum, long readahead) that calls runBlockReaderLocalTest(test, checksum, readahead, 1);

Thank you, I fixed it. But below by link writes "Could not update commit status". Should I create new PR or it is enough?

refacted tests decoration

hadoop-yetus · 2020-05-16T07:11:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	docker	0m 1s	Docker command '/usr/bin/docker' is too old ( < API v 1.27.0).

Subsystem	Report/Notes
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/3/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

finished decoration of tests

hadoop-yetus · 2020-05-16T07:41:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	docker	0m 0s	Docker command '/usr/bin/docker' is too old ( < API v 1.27.0).

Subsystem	Report/Notes
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/4/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-05-18T13:30:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	docker	0m 0s	Docker command '/usr/bin/docker' is too old ( < API v 1.27.0).

Subsystem	Report/Notes
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2016/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jojochuang · 2020-05-18T14:03:42Z

No I think you're fine. You probably need to rebase the PR to use the new docker. But I'll merge this one.

Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job. (cherry picked from commit 86e6aa8)

This reverts commit 86e6aa8.

This reverts commit ad9a6a0.

(cherry picked from commit 2abcf77)

(cherry picked from commit 2abcf77) Conflicts: hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java Change-Id: I257542e0efa18dd47212f240034af9520d88a934

Added parameter dfs.client.short.circuit.num improving HDFS-client's …

4e96f13

…massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

pustota2009 mentioned this pull request May 13, 2020

HDFS-15202 Boost short circuit cache #1884

Closed

Update DfsClientConf.java

7d723fc

fixed codestyle

jojochuang approved these changes May 15, 2020

View reviewed changes

Update TestBlockReaderLocal.java

5a1b8c9

refacted tests decoration

Update TestBlockReaderLocal.java

91c3af3

finished decoration of tests

jojochuang merged commit 86e6aa8 into apache:trunk May 18, 2020

asfgit pushed a commit that referenced this pull request May 18, 2020

Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)"

4525292

This reverts commit 86e6aa8.

asfgit pushed a commit that referenced this pull request May 18, 2020

HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)

2abcf77

asfgit pushed a commit that referenced this pull request May 18, 2020

Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)"

289f8ac

This reverts commit ad9a6a0.

asfgit pushed a commit that referenced this pull request May 18, 2020

HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)

240cba7

(cherry picked from commit 2abcf77)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15202 Boost short circuit cache (rebase PR-1884) #2016

HDFS-15202 Boost short circuit cache (rebase PR-1884) #2016

Uh oh!

pustota2009 commented May 13, 2020

Uh oh!

hadoop-yetus commented May 13, 2020

Uh oh!

jojochuang commented May 13, 2020

Uh oh!

hadoop-yetus commented May 13, 2020

Uh oh!

jojochuang left a comment

Uh oh!

jojochuang May 15, 2020

Uh oh!

pustota2009 May 16, 2020

Uh oh!

hadoop-yetus commented May 16, 2020

Uh oh!

hadoop-yetus commented May 16, 2020

Uh oh!

hadoop-yetus commented May 18, 2020

Uh oh!

jojochuang commented May 18, 2020

Uh oh!

Uh oh!

HDFS-15202 Boost short circuit cache (rebase PR-1884) #2016

HDFS-15202 Boost short circuit cache (rebase PR-1884) #2016

Uh oh!

Conversation

pustota2009 commented May 13, 2020

Uh oh!

hadoop-yetus commented May 13, 2020

Uh oh!

jojochuang commented May 13, 2020

Uh oh!

hadoop-yetus commented May 13, 2020

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang May 15, 2020

Choose a reason for hiding this comment

Uh oh!

pustota2009 May 16, 2020

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented May 16, 2020

Uh oh!

hadoop-yetus commented May 16, 2020

Uh oh!

hadoop-yetus commented May 18, 2020

Uh oh!

jojochuang commented May 18, 2020

Uh oh!

Uh oh!