HADOOP-19217. Introduce getTrashPolicy to FileSystem API

[ivandika3]

Description of PR

JIRA: https://issues.apache.org/jira/browse/HADOOP-19217

Hadoop FileSystem supports multiple FileSystem implementations awareness (e.g. client is aware of both hdfs:// and ofs:// protocols).

However, it seems that currently Hadoop TrashPolicy remains the same regardless of the URI scheme. The TrashPolicy is governed by "fs.trash.classname" configuration and stays the same regardless of the FileSystem implementation. For example, HDFS defaults to TrashPolicyDefault and Ozone defaults to TrashPolicyOzone, but only one will be picked since the the configuration will be overwritten by the other.

Therefore, I propose to tie the TrashPolicy implementation to each FileSystem implementation by introducing a new FileSystem#getTrashPolicy interface. TrashPolicy#getInstance can call FileSystem#getTrashPolicy to get the appropriate TrashPolicy.

How was this patch tested?

Unit and contract tests (HDFS and LocalFS).

Disclosure: FileSystem.md part was initially generated by AI, but majority of it was updated. The other implementations are hand-coded.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	6m 51s		Maven dependency ordering for branch
+1 💚	mvninstall	17m 38s		trunk passed
+1 💚	compile	10m 6s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	9m 57s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 41s		trunk passed
+1 💚	mvnsite	1m 59s		trunk passed
+1 💚	javadoc	1m 33s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 22s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	1m 40s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in trunk has 448 extant spotbugs warnings.
-1 ❌	spotbugs	2m 12s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html	hadoop-hdfs-project/hadoop-hdfs in trunk has 291 extant spotbugs warnings.
+1 💚	shadedclient	18m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 30s		the patch passed
+1 💚	compile	9m 47s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	9m 47s		the patch passed
+1 💚	compile	9m 56s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	9m 56s		the patch passed
-1 ❌	blanks	0m 1s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	1m 44s	/results-checkstyle-root.txt	root: The patch generated 28 new + 248 unchanged - 0 fixed = 276 total (was 248)
+1 💚	mvnsite	1m 59s		the patch passed
+1 💚	javadoc	1m 26s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 24s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	4m 1s		the patch passed
+1 💚	shadedclient	17m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	16m 29s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
-1 ❌	unit	168m 25s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		308m 15s

Reason	Tests
Failed junit tests	hadoop.fs.contract.localfs.TestLocalFSContractTrash
	hadoop.security.ssl.TestDelegatingSSLSocketFactory
	hadoop.fs.contract.hdfs.TestHDFSContractTrash
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8063/2/artifact/out/Dockerfile
GITHUB PR	#8063
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 438b5dd4e148 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 29c99cb
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8063/2/testReport/
Max. process+thread count	3971 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8063/2/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 25s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	7m 19s		Maven dependency ordering for branch
+1 💚	mvninstall	16m 34s		trunk passed
+1 💚	compile	8m 40s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	8m 46s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 43s		trunk passed
+1 💚	mvnsite	2m 3s		trunk passed
+1 💚	javadoc	1m 27s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 27s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	spotbugs	1m 39s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in trunk has 448 extant spotbugs warnings.
-1 ❌	spotbugs	2m 0s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html	hadoop-hdfs-project/hadoop-hdfs in trunk has 291 extant spotbugs warnings.
+1 💚	shadedclient	16m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 26s		the patch passed
+1 💚	compile	8m 14s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	8m 14s		the patch passed
+1 💚	compile	8m 37s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	8m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 34s	/results-checkstyle-root.txt	root: The patch generated 28 new + 248 unchanged - 0 fixed = 276 total (was 248)
+1 💚	mvnsite	2m 2s		the patch passed
+1 💚	javadoc	1m 28s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	3m 51s		the patch passed
+1 💚	shadedclient	16m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	16m 27s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
-1 ❌	unit	174m 4s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		305m 17s

Reason	Tests
Failed junit tests	hadoop.security.ssl.TestDelegatingSSLSocketFactory
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8063/4/artifact/out/Dockerfile
GITHUB PR	#8063
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux c4eea5429b3b 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 5af2d3b
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8063/4/testReport/
Max. process+thread count	4212 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8063/4/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[slfan1989]

@ayushtkn @KeeProMise Could you please review this PR? Thank you very much!

[ivandika3]

@steveloughran Please help to take a look when you are available. Thanks in advance.

[ayushtkn]

curious where we will use it, adding this as a FileSystem API looks overkill to me, if it required for some particular use case, maybe part of getServerDefaults()?

[ivandika3]

@ayushtkn Thanks for taking a look.

curious where we will use it

The problem that we encountered is that we have a client that can access both hdfs:// and ofs:// but TrashPolicyDefault and OzoneTrashPolicy is not compatible (TrashPolicyDefault was changed that cause issues when applied to Ozone). Despite setting "fs.trash.classname" in both HDFS and Ozone, only one will be picked (usually TrashPolicyDefault) which causes user not being able to move files to trash in Ozone.

Other related works like HADOOP-18013 and HADOOP-18893 approached it by using per-scheme configuration (e.g. fs.s3a.trash.classname)

adding this as a FileSystem API looks overkill to me, if it required for some particular use case, maybe part of getServerDefaults()?

I am not aware with getServerDefaults and it seems we can move the TrashPolicy as part of getServerDefaults if needed. However, I feel adding a new FileSystem API is more explicit which also allows us to specify the trash behavior in FileSystem.md. The default behavior is also backward compatible and will be transparent to users.