HDFS-16470. Make HDFS find tool cross platform #4076

GauthamBanasandra · 2022-03-16T18:15:28Z

Description of PR

The source files for hdfs_find uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make these tools cross platform.

How was this patch tested?

hdfs_find

I ran hdfs_find with root, it does a traversal of the whole filesystem from the root and displays the same.

$ ./hdfs_find /

I then ran the same command with the -m option which limits the depth of the traversal.

$ ./hdfs_find / -m 2

I ran the tool with the -n option, specifying the pattern of the file to find.

$ ./hdfs_find / -n *.txt

I then ran the tool with both the options - limiting the traversal depth and by the pattern to find.

$ ./hdfs_find / -m 2 -n *.txt

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

* Fixed some bugs in the process.

hadoop-yetus · 2022-03-16T21:00:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	35m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 31s		trunk passed
+1 💚	compile	3m 21s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	shadedclient	57m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	3m 15s		the patch passed
+1 💚	cc	3m 15s		the patch passed
+1 💚	golang	3m 15s		the patch passed
+1 💚	javac	3m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	shadedclient	18m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 14s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		150m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/artifact/out/Dockerfile
GITHUB PR	#4076
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux c48b38b504ef 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e425469`
Default Java	Red Hat, Inc.-1.8.0_322-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/testReport/
Max. process+thread count	734 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-16T23:07:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 4s		trunk passed
+1 💚	compile	3m 38s		trunk passed
+1 💚	mvnsite	0m 47s		trunk passed
+1 💚	shadedclient	46m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	3m 25s		the patch passed
+1 💚	cc	3m 25s		the patch passed
+1 💚	golang	3m 25s		the patch passed
+1 💚	javac	3m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 27s		the patch passed
+1 💚	shadedclient	19m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 42s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		126m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/artifact/out/Dockerfile
GITHUB PR	#4076
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 589d3837fa92 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e425469`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/testReport/
Max. process+thread count	742 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-17T01:18:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 50s		trunk passed
+1 💚	compile	3m 5s		trunk passed
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	shadedclient	55m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	3m 1s		the patch passed
+1 💚	cc	3m 1s		the patch passed
+1 💚	golang	3m 1s		the patch passed
+1 💚	javac	3m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 17s		the patch passed
+1 💚	shadedclient	25m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 52s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		131m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/artifact/out/Dockerfile
GITHUB PR	#4076
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux a34096bef274 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e425469`
Default Java	Debian-11.0.14+9-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/testReport/
Max. process+thread count	576 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-03-17T03:10:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 23s		trunk passed
+1 💚	compile	3m 27s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	3m 27s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 27s		trunk passed
+1 💚	shadedclient	48m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	3m 23s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	cc	3m 23s		the patch passed
+1 💚	golang	3m 23s		the patch passed
+1 💚	javac	3m 23s		the patch passed
+1 💚	compile	3m 24s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	3m 24s		the patch passed
+1 💚	golang	3m 24s		the patch passed
+1 💚	javac	3m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	shadedclient	19m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 46s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		111m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/artifact/out/Dockerfile
GITHUB PR	#4076
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux a5744a1c8c6b 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e425469`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/testReport/
Max. process+thread count	544 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4076/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* The source files for hdfs_find uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. * Thus, we need to replace getopt with boost::program_options to make hdfs_find cross platform.

GauthamBanasandra added 6 commits March 16, 2022 23:38

Add skeleton for hdfs-find

38ba6d6

Make hdfs-find cross platform

a35daf5

Add unit test for hdfs_find

de0a438

* Fixed some bugs in the process.

Add documentation

a37dedf

Remove redundant file

9a8d379

Fix typos and whitespace

e425469

GauthamBanasandra self-assigned this Mar 16, 2022

GauthamBanasandra marked this pull request as ready for review March 17, 2022 17:25

GauthamBanasandra requested a review from goiri March 17, 2022 17:25

GauthamBanasandra added the HDFS label Mar 17, 2022

goiri approved these changes Mar 17, 2022

View reviewed changes

GauthamBanasandra merged commit a631f45 into apache:trunk Mar 18, 2022

GauthamBanasandra deleted the find-x-platform branch March 18, 2022 16:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16470. Make HDFS find tool cross platform #4076

HDFS-16470. Make HDFS find tool cross platform #4076

GauthamBanasandra commented Mar 16, 2022 •

edited

Loading

hadoop-yetus commented Mar 16, 2022

hadoop-yetus commented Mar 16, 2022

hadoop-yetus commented Mar 17, 2022

hadoop-yetus commented Mar 17, 2022

HDFS-16470. Make HDFS find tool cross platform #4076

HDFS-16470. Make HDFS find tool cross platform #4076

Conversation

GauthamBanasandra commented Mar 16, 2022 • edited Loading

Description of PR

How was this patch tested?

hdfs_find

For code changes:

hadoop-yetus commented Mar 16, 2022

hadoop-yetus commented Mar 16, 2022

hadoop-yetus commented Mar 17, 2022

hadoop-yetus commented Mar 17, 2022

GauthamBanasandra commented Mar 16, 2022 •

edited

Loading