HDFS-16749. RBF: Gets the wrong directory information from Trash #6317

LiuGuH · 2023-12-01T14:35:57Z

Description of PR

As described in HDFS-16749.

Purpose ：[RBF] Fix ls user trash path returns mountpoint path that real nameservice's trash path is not exists

There are two nameservices ns0 ns1, and ns0 is the default nameservice.

(1) Add moutTable

/home/data -> (ns0, /home/data)
/data1/test1 -> (ns1, /data1/test1 )
/data2/test2 -> (ns1, /data2/test2 )

(2) add file

/home/data/file1
/data1/test1/file1

(3) mv files to trash

ns0:   /user/test-user/.Trash/Current/home/data/file1
ns1:   /user/test-user/.Trash/Current/data1/test1/file1

(4) client via DFSRouter: hdfs dfs -ls /user/test-user/.Trash/Current ,this will return

/user/test-user/.Trash/Current/home
/user/test-user/.Trash/Current/data1
/user/test-user/.Trash/Current/data2  (But this in fact not exist in ns1)

So, this PR is for this purpose, if a namservice that mount point dir in trash is not really exists, it should be not display in user trash view via DFSRouter

…rvice's trash path is not exists

slfan1989 · 2023-12-01T15:02:36Z

@LiuGuH Thank you for your contribution! Could you please provide more details about your proposed modification?

slfan1989 · 2023-12-01T15:14:32Z

...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterTrash.java

@@ -276,7 +276,7 @@ public void testMultipleMountPoint() throws IOException,

    // Client user see global trash view， wo should see all three mount point.
    FileStatus[] fileStatuses = fs.listStatus(new Path("/user/test-trash/.Trash/Current/"));
-    assertEquals(3, fileStatuses.length);
+    assertEquals(2, fileStatuses.length);


Why does the size change?

I add description in Description of PR. Thanks

I modified the description and let's wait for other partners' suggestions. My personal suggestion is that we'd better solve the issue with smaller changes. (#5039)

With #5039 , it will lead to user will not see any other nameservices trash path expect where the user home is mounted nameservices. It will break https://issues.apache.org/jira/browse/HDFS-16024 purpose.

hadoop-yetus · 2023-12-01T17:02:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 52s		trunk passed
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 43s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 21s		trunk passed
+1 💚	shadedclient	32m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 30s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 17s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 3 unchanged - 0 fixed = 6 total (was 3)
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
-1 ❌	spotbugs	1m 23s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	32m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	22m 2s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		144m 57s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	org.apache.hadoop.hdfs.server.federation.resolver.FileSubclusterResolver.getMountPointsWithSrc(String, SortedMap) makes inefficient use of keySet iterator instead of entrySet iterator At FileSubclusterResolver.java:of keySet iterator instead of entrySet iterator At FileSubclusterResolver.java:[line 184]
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpc
	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/1/artifact/out/Dockerfile
GITHUB PR	#6317
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 575e6ad58f8a 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c0b98fc`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/1/testReport/
Max. process+thread count	3448 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2023-12-02T04:06:10Z

@zhangxiping1 I see that you created HDFS-16749, pr #5039, and @LiuGuH also submitted a PR. Can we compare and explain the difference between the two prs?

hadoop-yetus · 2023-12-02T06:08:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 18s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 29s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 41s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 31s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 19s		trunk passed
+1 💚	shadedclient	32m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 28s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 18s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 18s		the patch passed
+1 💚	shadedclient	32m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	21m 39s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		155m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/2/artifact/out/Dockerfile
GITHUB PR	#6317
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux deaf1fa46cc3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e73bf6d`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/2/testReport/
Max. process+thread count	2840 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-12-02T08:42:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 21s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 19s		trunk passed
+1 💚	shadedclient	32m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 18s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3)
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 19s		the patch passed
+1 💚	shadedclient	32m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	21m 44s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		143m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/3/artifact/out/Dockerfile
GITHUB PR	#6317
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 289fa402ff9b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `41aeea7`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/3/testReport/
Max. process+thread count	2841 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6317/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2023-12-03T11:15:47Z

Thanks @LiuGuH for your works.

(4) client via DFSRouter: hdfs dfs -ls /user/test-user/.Trash/Current ,this will return

I am confused why it returns wrong subdirs now? Thanks.

zhangxiping1 · 2023-12-04T03:47:32Z

@zhangxiping1 I see that you created HDFS-16749, pr #5039, and @LiuGuH also submitted a PR. Can we compare and explain the difference between the two prs?
Is the same problem, returning the extra entry is the Trash prefix with mount point information(top subdirectory)

LiuGuH · 2023-12-04T08:03:01Z

(4) client via DFSRouter: hdfs dfs -ls /user/test-user/.Trash/Current ,this will return

I am confused why it returns wrong subdirs now? Thanks.

getListing can be separated into two parts
(1) getListingInt

subtractTrashCurrentPath("/user/test-user/.Trash/Current") will get "" as moint point in default nameservices ns0, then construct monit point
/user/test-user/.Trash/Current -> (ns0,/user/test-user/.Trash/Current)
And then execute getListingInt.

(2) subclusterResolver.getMountPoints(src)
getMountPoints("/user/test-user/.Trash/Current") will get all children of "" , this will return all mount points

home
data1
data2

and then it be display

/user/test-user/.Trash/Current/home
/user/test-user/.Trash/Current/data1
/user/test-user/.Trash/Current/data2

getMountPoints is necessary because we will not see all subtrash dirs without it.
The problem is that these mountpoint dir maybe not exists if user does not delelt any dir or file into trash. And now it is simple display all mount point.

My way of dealing with it is that for every child path，found its mount point ,and getFileInfo with it . If it is really exits, display it. @Hexiaoqiao @slfan1989

[RBF] Fix ls user trash path returns mountpoint path that real namese…

c0b98fc

…rvice's trash path is not exists

github-actions bot added HDFS trunk RBF labels Dec 1, 2023

slfan1989 reviewed Dec 1, 2023

View reviewed changes

fix test case

e73bf6d

fix blanks

41aeea7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16749. RBF: Gets the wrong directory information from Trash #6317

HDFS-16749. RBF: Gets the wrong directory information from Trash #6317

LiuGuH commented Dec 1, 2023 •

edited by slfan1989

Loading

Uh oh!

slfan1989 commented Dec 1, 2023

Uh oh!

slfan1989 Dec 1, 2023

Uh oh!

LiuGuH Dec 2, 2023

Uh oh!

slfan1989 Dec 2, 2023 •

edited

Loading

Uh oh!

LiuGuH Dec 2, 2023

Uh oh!

hadoop-yetus commented Dec 1, 2023

Uh oh!

slfan1989 commented Dec 2, 2023 •

edited

Loading

Uh oh!

hadoop-yetus commented Dec 2, 2023

Uh oh!

hadoop-yetus commented Dec 2, 2023

Uh oh!

Hexiaoqiao commented Dec 3, 2023

Uh oh!

zhangxiping1 commented Dec 4, 2023

Uh oh!

LiuGuH commented Dec 4, 2023 •

edited

Loading

Uh oh!

Uh oh!

HDFS-16749. RBF: Gets the wrong directory information from Trash #6317

Are you sure you want to change the base?

HDFS-16749. RBF: Gets the wrong directory information from Trash #6317

Conversation

LiuGuH commented Dec 1, 2023 • edited by slfan1989 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Uh oh!

slfan1989 commented Dec 1, 2023

Uh oh!

slfan1989 Dec 1, 2023

Choose a reason for hiding this comment

Uh oh!

LiuGuH Dec 2, 2023

Choose a reason for hiding this comment

Uh oh!

slfan1989 Dec 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LiuGuH Dec 2, 2023

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Dec 1, 2023

Uh oh!

slfan1989 commented Dec 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Dec 2, 2023

Uh oh!

hadoop-yetus commented Dec 2, 2023

Uh oh!

Hexiaoqiao commented Dec 3, 2023

Uh oh!

zhangxiping1 commented Dec 4, 2023

Uh oh!

LiuGuH commented Dec 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

LiuGuH commented Dec 1, 2023 •

edited by slfan1989

Loading

slfan1989 Dec 2, 2023 •

edited

Loading

slfan1989 commented Dec 2, 2023 •

edited

Loading

LiuGuH commented Dec 4, 2023 •

edited

Loading