HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc. #6994

LeoLeeeeee · 2024-08-13T02:39:29Z

Description of PR

please see https://issues.apache.org/jira/browse/HDFS-17597

How was this patch tested?

new ut TestRouterAsyncSnapshot

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-08-13T04:36:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ HDFS-17531 Compile Tests _
+1 💚	mvninstall	32m 58s		HDFS-17531 passed
+1 💚	compile	0m 25s		HDFS-17531 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 22s		HDFS-17531 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 21s		HDFS-17531 passed
+1 💚	mvnsite	0m 31s		HDFS-17531 passed
+1 💚	javadoc	0m 32s		HDFS-17531 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 21s		HDFS-17531 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 57s		HDFS-17531 passed
+1 💚	shadedclient	20m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 23s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		the patch passed
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 51s		the patch passed
+1 💚	shadedclient	20m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	27m 19s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		115m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/1/artifact/out/Dockerfile
GITHUB PR	#6994
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 54e39f981022 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17531 / `591e648`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/1/testReport/
Max. process+thread count	3809 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

KeeProMise · 2024-08-13T13:53:35Z

...s-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAsyncSnapshot.java

+          return invokedResult.replaceFirst(loc.getDest(), loc.getSrc());
+        });
+    }
+    return AsyncUtil.asyncReturn(String.class);


@LeoLeeeeee Hi, thank you for your contribution. I think AsyncUtil can be deleted here. You can use asyncReturn(String.class) directly.

KeeProMise · 2024-08-13T13:55:05Z

...s-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAsyncSnapshot.java

+    asyncApply((ApplyFunction<Map<FederationNamespaceInfo, SnapshottableDirectoryStatus[]>,
+        SnapshottableDirectoryStatus[]>)
+        ret -> RouterRpcServer.merge(ret, SnapshottableDirectoryStatus.class));
+    return AsyncUtil.asyncReturn(SnapshottableDirectoryStatus[].class);


Same as above.

KeeProMise · 2024-09-27T11:05:25Z

Hi, @LeoLeeeeee HDFS-17545 has already been merged into HDFS-17531, please rebase your development branch using HDFS-17531.

LeoLeeeeee · 2024-09-27T11:43:02Z

Hi, @LeoLeeeeee HDFS-17545 has already been merged into HDFS-17531, please rebase your development branch using HDFS-17531.

@KeeProMise Ok, recommit, please take a look again.

hadoop-yetus · 2024-09-27T13:27:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ HDFS-17531 Compile Tests _
+1 💚	mvninstall	33m 38s		HDFS-17531 passed
+1 💚	compile	0m 26s		HDFS-17531 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 21s		HDFS-17531 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 19s		HDFS-17531 passed
+1 💚	mvnsite	0m 24s		HDFS-17531 passed
+1 💚	javadoc	0m 27s		HDFS-17531 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 20s		HDFS-17531 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 53s		HDFS-17531 passed
+1 💚	shadedclient	21m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 50s		the patch passed
+1 💚	shadedclient	22m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	29m 37s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		115m 5s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/2/artifact/out/Dockerfile
GITHUB PR	#6994
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 85c883ace502 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17531 / `428c52d`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/2/testReport/
Max. process+thread count	3244 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-09-27T13:35:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ HDFS-17531 Compile Tests _
+1 💚	mvninstall	34m 15s		HDFS-17531 passed
+1 💚	compile	0m 25s		HDFS-17531 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 22s		HDFS-17531 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 19s		HDFS-17531 passed
+1 💚	mvnsite	0m 24s		HDFS-17531 passed
+1 💚	javadoc	0m 27s		HDFS-17531 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 19s		HDFS-17531 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 53s		HDFS-17531 passed
+1 💚	shadedclient	21m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 20s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 23s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 15s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	0m 49s		the patch passed
+1 💚	shadedclient	23m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	27m 5s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		114m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/3/artifact/out/Dockerfile
GITHUB PR	#6994
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 705019f217c8 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HDFS-17531 / `50a13b3`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/3/testReport/
Max. process+thread count	3704 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6994/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

KeeProMise

LGTM! @Hexiaoqiao Hi, if you have time, please help to take a look at this PR, thanks!

Hexiaoqiao · 2024-10-15T09:37:15Z

@LeoLeeeeee @KeeProMise Thanks for your works. LGTM. +1 from my side. One thing as #6983 mentioned, do we need to move async RPC implement under single package? Thanks.

KeeProMise · 2024-10-15T09:42:57Z

@LeoLeeeeee @KeeProMise Thanks for your works. LGTM. +1 from my side. One thing as #6983 mentioned, do we need to move async RPC implement under single package? Thanks.

@Hexiaoqiao Thanks, I also have this idea. I will commit a PR in the future and put these PRs together with the previous PRs under an async package.

Hexiaoqiao · 2024-10-16T02:55:33Z

make sense to me. +1.

KeeProMise · 2024-10-16T05:18:47Z

@LeoLeeeeee Thanks for your contribution!
@Hexiaoqiao Thanks for your review!

). Contributed by Wenqi Li. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

…Contributed by Wenqi Li. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

). Contributed by Wenqi Li. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

github-actions bot added HDFS trunk RBF labels Aug 13, 2024

KeeProMise reviewed Aug 13, 2024

View reviewed changes

KeeProMise mentioned this pull request Sep 11, 2024

HDFS-17545. [ARR] router async rpc client. #6871

Merged

4 tasks

Hexiaoqiao force-pushed the HDFS-17531 branch from 4e0b405 to a591902 Compare September 12, 2024 07:44

Wenqi Li and others added 2 commits September 27, 2024 19:26

[ARR] RouterSnapshot supports asynchronous rpc.

7d3deb1

[ARR] RouterSnapshot supports asynchronous rpc.

428c52d

LeoLeeeeee force-pushed the HDFS-17597 branch from 591e648 to 428c52d Compare September 27, 2024 11:31

delete AsyncUtil

50a13b3

KeeProMise requested a review from Hexiaoqiao September 27, 2024 13:39

KeeProMise approved these changes Sep 27, 2024

View reviewed changes

KeeProMise mentioned this pull request Oct 8, 2024

HDFS-17595. [ARR] ErasureCoding supports asynchronous rpc. #6983

Merged

Hexiaoqiao approved these changes Oct 16, 2024

View reviewed changes

KeeProMise merged commit 7d4a628 into apache:HDFS-17531 Oct 16, 2024
4 checks passed

KeeProMise mentioned this pull request Jan 22, 2025

HDFS-17531. RBF: Aynchronous router RPC. #7308

Closed

4 tasks

KeeProMise pushed a commit that referenced this pull request Feb 14, 2025

HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc. (#6994). …

37c7cdd

…Contributed by Wenqi Li. Reviewed-by: Jian Zhang <keepromise@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc. #6994

HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc. #6994

Uh oh!

LeoLeeeeee commented Aug 13, 2024 •

edited

Loading

Uh oh!

hadoop-yetus commented Aug 13, 2024

Uh oh!

KeeProMise Aug 13, 2024

Uh oh!

KeeProMise Aug 13, 2024

Uh oh!

KeeProMise commented Sep 27, 2024

Uh oh!

LeoLeeeeee commented Sep 27, 2024

Uh oh!

hadoop-yetus commented Sep 27, 2024

Uh oh!

hadoop-yetus commented Sep 27, 2024

Uh oh!

KeeProMise left a comment

Uh oh!

Hexiaoqiao commented Oct 15, 2024

Uh oh!

KeeProMise commented Oct 15, 2024

Uh oh!

Hexiaoqiao commented Oct 16, 2024

Uh oh!

Uh oh!

KeeProMise commented Oct 16, 2024

Uh oh!

Uh oh!

HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc. #6994

HDFS-17597. [ARR] RouterSnapshot supports asynchronous rpc. #6994

Uh oh!

Conversation

LeoLeeeeee commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Aug 13, 2024

Uh oh!

KeeProMise Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

KeeProMise Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

KeeProMise commented Sep 27, 2024

Uh oh!

LeoLeeeeee commented Sep 27, 2024

Uh oh!

hadoop-yetus commented Sep 27, 2024

Uh oh!

hadoop-yetus commented Sep 27, 2024

Uh oh!

KeeProMise left a comment

Choose a reason for hiding this comment

Uh oh!

Hexiaoqiao commented Oct 15, 2024

Uh oh!

KeeProMise commented Oct 15, 2024

Uh oh!

Hexiaoqiao commented Oct 16, 2024

Uh oh!

Uh oh!

KeeProMise commented Oct 16, 2024

Uh oh!

Uh oh!

LeoLeeeeee commented Aug 13, 2024 •

edited

Loading