HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout #3080

liangxs · 2021-06-07T05:03:27Z

…imeout

hadoop-yetus · 2021-06-07T08:04:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 27s		trunk passed
+1 💚	compile	20m 56s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	18m 3s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 8s		trunk passed
+1 💚	mvnsite	1m 34s		trunk passed
+1 💚	javadoc	1m 6s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 40s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	2m 24s		trunk passed
+1 💚	shadedclient	15m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	20m 9s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	20m 9s		the patch passed
+1 💚	compile	18m 15s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	18m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 7s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 6 new + 1 unchanged - 7 fixed = 7 total (was 8)
+1 💚	mvnsite	1m 35s		the patch passed
+1 💚	javadoc	1m 6s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌	spotbugs	2m 36s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌	shadedclient	15m 28s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	18m 18s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		180m 2s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Redundant nullcheck of selInfo, which is known to be non-null in org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SelectableChannel) Redundant null check at SocketIOWithTimeout.java:is known to be non-null in org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SelectableChannel) Redundant null check at SocketIOWithTimeout.java:[line 394]
Failed junit tests	hadoop.ipc.TestRPCWaitForProxy
	hadoop.ipc.TestMiniRPCBenchmark
	hadoop.ipc.TestProtoBufRpc
	hadoop.security.TestDoAsEffectiveUser
	hadoop.ipc.TestRPCCallBenchmark
	hadoop.ipc.metrics.TestRpcMetrics
	hadoop.ipc.TestIPC
	hadoop.ipc.TestIPCServerResponder
	hadoop.ipc.TestRPC
	hadoop.ipc.TestMultipleProtocolServer
	hadoop.ipc.TestSaslRPC
	hadoop.ipc.TestServer
	hadoop.ha.TestHealthMonitorWithDedicatedHealthAddress
	hadoop.ipc.TestAsyncIPC
	hadoop.ipc.TestRPCServerShutdown
	hadoop.ipc.TestProtoBufRpcServerHandoff
	hadoop.net.TestNetUtils
	hadoop.net.TestSocketIOWithTimeout
	hadoop.ipc.TestProtoBufRPCCompatibility
	hadoop.ipc.TestRpcServerHandoff
	hadoop.ha.TestHealthMonitor

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/1/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 7157121adbf9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `34c264c`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/1/testReport/
Max. process+thread count	1312 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-06-07T11:53:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 14s		trunk passed
+1 💚	compile	20m 48s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	18m 12s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 37s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	2m 29s		trunk passed
+1 💚	shadedclient	15m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	20m 10s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	20m 10s		the patch passed
+1 💚	compile	18m 14s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	18m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 6s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 6 new + 1 unchanged - 7 fixed = 7 total (was 8)
+1 💚	mvnsite	1m 33s		the patch passed
+1 💚	javadoc	1m 6s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	2m 34s		the patch passed
+1 💚	shadedclient	15m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 3s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		178m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/2/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux a918fffcfe9a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e02750e`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/2/testReport/
Max. process+thread count	1942 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

liangxs · 2021-06-08T12:00:58Z

Could someone please review this PR?

liangxs · 2021-06-09T10:45:41Z

@jojochuang Can you kindly review this PR?
Thanks.

liangxs · 2021-06-15T12:59:01Z

I tested the performance of the trunk version and the optimized version.

Test Case

The test steps are as follow:

Start a netty-echo-server with 100 service ports.

private static void startServer() throws Exception {
  ChannelHandler serverHandler = new EchoHandler();
  for (int i = 0; i < 100; ++i) {
    ServerBootstrap b = new ServerBootstrap();
    b.group(new NioEventLoopGroup(1), new NioEventLoopGroup(2))
        .channel(NioServerSocketChannel.class)
        .option(ChannelOption.SO_BACKLOG, 512)
        .childOption(ChannelOption.SO_TIMEOUT, timeout)
        .childOption(ChannelOption.TCP_NODELAY, true)
        .childHandler(serverHandler);
    ChannelFuture f = b.bind(host, port + i).sync();
  }
  Thread.sleep(Integer.MAX_VALUE);
}

Start a hadoop socket client with multi-threads.
These threads connect to the netty-echo-server's ports by a round robin manner.

private static void startClient(int threadCnt) throws Exception {
  SocketFactory factory = new StandardSocketFactory();
  Thread[] tArray = new Thread[threadCnt];
  CountDownLatch latch = new CountDownLatch(threadCnt);
  for (int i = 0; i < threadCnt; ++i) {
    final int curPort = port + (i % 100);  // round robin
    Thread t = new Thread(() -> {
      try {
        Socket socket = factory.createSocket();
        socket.setTcpNoDelay(true);
        socket.setKeepAlive(false);
        NetUtils.connect(socket, new java.net.InetSocketAddress(host, curPort), timeout);
        socket.setSoTimeout(timeout);
    ...
    ...

Each client thread send-to/recv-from its corresponding netty-server port 1024 times, with 256-byte data each time.

      InputStream inStream = NetUtils.getInputStream(socket);
      OutputStream outStream = NetUtils.getOutputStream(socket, timeout);
      DataInputStream in = new DataInputStream(new BufferedInputStream(inStream));
      DataOutputStream out = new DataOutputStream(new BufferedOutputStream(outStream));

      byte[] buf = new byte[256];
      for (int j = 0; j < 1024; ++j) {
        out.write(buf);
        out.flush();
        in.readFully(buf);
      }

print the total cost.

Code project: https://github.com/liangxs/test-HADOOP-17749

Test Result

The test result is as follow:

| client thread count | 100 | 200 | 400  | 800  | 1200 | 1600 | 2000 | 2400 | 2800 |
|---------------------|-----|-----|------|------|------|------|------|------|------|
| trunk (millis)      | 351 | 609 | 1058 | 2024 | 2907 | 3882 | 5062 | 5675 | 7117 |
| optimized (millis)  | 253 | 438 |  799 | 1167 | 1561 | 2422 | 2784 | 2813 | 3371 |
| improved            | 38% | 39% |  32% |  70% |  86% |  60% |  82% | 102% | 111% |

ps, I used two test machines with same hardware and software configuration and on a same rack:

$ lscpu
Architecture:          x86_64
CPU(s):                56
On-line CPU(s) list:   0-55
Thread(s) per core:    2
Core(s) per socket:    14
Socket(s):             2
NUMA node(s):          2
Model name:            Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
CPU MHz:               2401.000

$ free -g
              total        used        free      shared  buff/cache   available
Mem:             62           7          37           0          17          54

$ lspci | grep Ethernet
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

$ uname -r
3.10.107-1

liangxs · 2021-06-21T09:54:12Z

I execute this command in my product environment:

for i in `seq 1 20`; do jstack 23730 > d.jstack.`date +%H%M%S` ; sleep 1; done &> /dev/null

20 jstack files are generated:

Most of the lock contentions are about SelectPool:

ferhui · 2021-06-22T08:01:45Z

@liangxs Thanks for contribution, performance looks pretty good.
This PR looks mostly good, could you please add unit test case?

…_contention

hadoop-yetus · 2021-06-22T13:35:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 44s		trunk passed
+1 💚	compile	20m 55s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	18m 14s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 36s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 25s		trunk passed
+1 💚	shadedclient	15m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	20m 17s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	20m 17s		the patch passed
+1 💚	compile	18m 16s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 6s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 6 new + 1 unchanged - 7 fixed = 7 total (was 8)
+1 💚	mvnsite	1m 33s		the patch passed
+1 💚	javadoc	1m 7s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 40s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 33s		the patch passed
+1 💚	shadedclient	15m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 58s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		176m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/4/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 1104423b17f0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `efd9d7c`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/4/testReport/
Max. process+thread count	1281 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-06-22T13:36:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 38s		trunk passed
+1 💚	compile	21m 11s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	18m 16s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 8s		trunk passed
+1 💚	mvnsite	1m 36s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 24s		trunk passed
+1 💚	shadedclient	15m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 55s		the patch passed
+1 💚	compile	20m 18s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	20m 18s		the patch passed
+1 💚	compile	18m 50s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 50s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 1s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 6 new + 1 unchanged - 7 fixed = 7 total (was 8)
+1 💚	mvnsite	1m 35s		the patch passed
+1 💚	javadoc	1m 5s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 36s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 29s		the patch passed
+1 💚	shadedclient	15m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 41s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 55s		The patch does not generate ASF License warnings.
		178m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/3/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 1fc4cf9c0e1f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `efd9d7c`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/3/testReport/
Max. process+thread count	2601 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-06-22T15:43:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 39s		trunk passed
+1 💚	compile	22m 54s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 13s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 35s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 26s		trunk passed
+1 💚	shadedclient	15m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 58s		the patch passed
+1 💚	compile	20m 55s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	20m 55s		the patch passed
+1 💚	compile	18m 18s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 5s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 4 new + 6 unchanged - 7 fixed = 10 total (was 13)
+1 💚	mvnsite	1m 33s		the patch passed
+1 💚	javadoc	1m 7s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 34s		the patch passed
+1 💚	shadedclient	15m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 2s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		180m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/5/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 0df06da28b40 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4fa9c7e`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/5/testReport/
Max. process+thread count	1500 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

liangxs · 2021-06-23T02:45:15Z

@ferhui Thanks for your suggestion, I added unit test.

ferhui · 2021-06-23T06:15:57Z

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java

+      if (!trimming.compareAndSet(false, true)) {
+        return;
+      }
+      if (now - lastTrimTime < IDLE_TIMEOUT / 2) {


Why is it IDLE_TIMEOUT/2 here?
Maybe idle time is near 1.5 times IDLE_TIMEOUT in extreme cases。

@ferhui I remove this check in new commit.

ferhui · 2021-06-23T06:20:35Z

And please see the checkstyle reports

hadoop-yetus · 2021-06-23T10:40:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	28m 0s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	29m 12s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	24m 33s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 11s		trunk passed
+1 💚	mvnsite	1m 54s		trunk passed
+1 💚	javadoc	1m 14s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 48s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 48s		trunk passed
+1 💚	shadedclient	21m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 12s		the patch passed
+1 💚	compile	21m 8s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	21m 8s		the patch passed
+1 💚	compile	18m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 6s		hadoop-common-project/hadoop-common: The patch generated 0 new + 6 unchanged - 7 fixed = 6 total (was 13)
+1 💚	mvnsite	1m 34s		the patch passed
+1 💚	javadoc	1m 8s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 33s		the patch passed
+1 💚	shadedclient	15m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 4s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		196m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/6/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 3b8b7bf7739f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f0c98c2`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/6/testReport/
Max. process+thread count	1211 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ferhui · 2021-06-28T01:06:08Z

@aajisaka @jojochuang Could you please help review this? performance improvement looks good

aajisaka

The patch mostly looks good to me. Thank you @liangxs.

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java

hadoop-yetus · 2021-07-05T05:18:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 4s		trunk passed
+1 💚	compile	21m 41s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 0s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 7s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 47s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 25s		trunk passed
+1 💚	shadedclient	15m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 53s		the patch passed
+1 💚	compile	20m 10s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	20m 10s		the patch passed
+1 💚	compile	18m 11s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 11s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 5s		hadoop-common-project/hadoop-common: The patch generated 0 new + 6 unchanged - 7 fixed = 6 total (was 13)
+1 💚	mvnsite	1m 34s		the patch passed
+1 💚	javadoc	1m 7s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 36s		the patch passed
+1 💚	shadedclient	15m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 4s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		177m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/7/artifact/out/Dockerfile
GITHUB PR	#3080
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux bf2c58d82d3f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e822825`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/7/testReport/
Max. process+thread count	1549 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3080/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ferhui · 2021-07-06T01:10:47Z

@liangxs Thanks for contribution. @aajisaka Thanks for review!
Merged to trunk.

liangxs · 2021-07-06T01:43:45Z

@ferhui @aajisaka Thanks very much for your reviews!

…imeout (#3080) (cherry picked from commit a5db683)

…imeout (apache#3080)

…cketIOWithTimeout (apache#3080) (cherry picked from commit a5db683) Change-Id: Ief8d9eb2700a340f2dea0a9fd4c1e8791bdce37c (cherry picked from commit 753eb1766eb67760d058a17ff31c399eaf2edd2a)

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithT…

34c264c

…imeout

trivial

e02750e

Merge branch 'trunk' into lock_contention

f091f37

liangxs and others added 3 commits June 22, 2021 18:37

Merge branch 'apache:trunk' into lock_contention

efd9d7c

add unit test

6eef870

Merge branch 'lock_contention' of github.com:liangxs/hadoop into lock…

4fa9c7e

…_contention

ferhui reviewed Jun 23, 2021

View reviewed changes

fix code style, and remove time check

f0c98c2

aajisaka requested changes Jul 3, 2021

View reviewed changes

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java Outdated Show resolved Hide resolved

...op-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java Show resolved Hide resolved

remove useless field

e822825

aajisaka approved these changes Jul 5, 2021

View reviewed changes

ferhui approved these changes Jul 6, 2021

View reviewed changes

ferhui merged commit a5db683 into apache:trunk Jul 6, 2021

asfgit pushed a commit that referenced this pull request Jul 7, 2021

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithT…

24b7808

…imeout (#3080) (cherry picked from commit a5db683)

asfgit pushed a commit that referenced this pull request Jul 7, 2021

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithT…

b7108cf

…imeout (#3080) (cherry picked from commit a5db683)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithT…

c651a30

…imeout (apache#3080)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout #3080

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout #3080

liangxs commented Jun 7, 2021

hadoop-yetus commented Jun 7, 2021

hadoop-yetus commented Jun 7, 2021

liangxs commented Jun 8, 2021

liangxs commented Jun 9, 2021

liangxs commented Jun 15, 2021

liangxs commented Jun 21, 2021

ferhui commented Jun 22, 2021

hadoop-yetus commented Jun 22, 2021

hadoop-yetus commented Jun 22, 2021

hadoop-yetus commented Jun 22, 2021

liangxs commented Jun 23, 2021

ferhui Jun 23, 2021

liangxs Jun 23, 2021

ferhui commented Jun 23, 2021

hadoop-yetus commented Jun 23, 2021

ferhui commented Jun 28, 2021

aajisaka left a comment

hadoop-yetus commented Jul 5, 2021

ferhui commented Jul 6, 2021

liangxs commented Jul 6, 2021

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout #3080

HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout #3080

Conversation

liangxs commented Jun 7, 2021

hadoop-yetus commented Jun 7, 2021

hadoop-yetus commented Jun 7, 2021

liangxs commented Jun 8, 2021

liangxs commented Jun 9, 2021

liangxs commented Jun 15, 2021

Test Case

Test Result

liangxs commented Jun 21, 2021

ferhui commented Jun 22, 2021

hadoop-yetus commented Jun 22, 2021

hadoop-yetus commented Jun 22, 2021

hadoop-yetus commented Jun 22, 2021

liangxs commented Jun 23, 2021

ferhui Jun 23, 2021

Choose a reason for hiding this comment

liangxs Jun 23, 2021

Choose a reason for hiding this comment

ferhui commented Jun 23, 2021

hadoop-yetus commented Jun 23, 2021

ferhui commented Jun 28, 2021

aajisaka left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Jul 5, 2021

ferhui commented Jul 6, 2021

liangxs commented Jul 6, 2021