HDFS-16474. Make HDFS tail tool cross platform #4157

GauthamBanasandra · 2022-04-10T15:09:33Z

Description of PR

The source files for hdfs_tail uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make these tools cross platform.

How was this patch tested?

hdfs_tail

I tested this on a Hadoop cluster setup locally in pseudo-distributed mode for various scenarios.
I ran hdfs_tail on a file whose size was more than 1 KB.

$ ./hdfs_tail ~/some-dir/Lorem-ipsum.txt

We note that it correctly displays the last 1 KB of data as verified with piping to wc.

I then ran this tool with the follow option.

$ ./hdfs_tail -f ~/some-dir/Lorem-ipsum.txt

It again displayed the same last 1 KB of the data in the file and I had to press Ctrl + C to exit the program.

I also tested it by giving a non-existent path.

$ ./hdfs_tail ~/some-non-existent-path

We note that it provides an appropriate error message and exits.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2022-04-10T19:10:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	47m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 30s		trunk passed
+1 💚	compile	3m 40s		trunk passed
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	shadedclient	69m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 38s		the patch passed
+1 💚	cc	3m 38s		the patch passed
+1 💚	golang	3m 38s		the patch passed
+1 💚	javac	3m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	22m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	93m 39s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		240m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/artifact/out/Dockerfile
GITHUB PR	#4157
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 1719e126d85a 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0cc0e06`
Default Java	Red Hat, Inc.-1.8.0_322-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/testReport/
Max. process+thread count	559 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-04-10T22:33:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	25m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 12s		trunk passed
+1 💚	compile	3m 50s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	shadedclient	52m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	3m 41s		the patch passed
+1 💚	cc	3m 41s		the patch passed
+1 💚	golang	3m 41s		the patch passed
+1 💚	javac	3m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	shadedclient	22m 37s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	94m 23s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		203m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/artifact/out/Dockerfile
GITHUB PR	#4157
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux b196e434c4e4 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0cc0e06`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/testReport/
Max. process+thread count	523 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-04-11T02:08:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	15m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 51s		trunk passed
+1 💚	compile	3m 31s		trunk passed
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	shadedclient	68m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 24s		the patch passed
+1 💚	cc	3m 24s		the patch passed
+1 💚	golang	3m 24s		the patch passed
+1 💚	javac	3m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	shadedclient	31m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	92m 7s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		214m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/artifact/out/Dockerfile
GITHUB PR	#4157
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 2eca3456a886 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0cc0e06`
Default Java	Debian-11.0.14+9-post-Debian-1deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/testReport/
Max. process+thread count	614 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-04-11T05:08:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 42s		trunk passed
+1 💚	compile	4m 9s		trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	compile	4m 8s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	shadedclient	56m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	4m 6s		the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	cc	4m 6s		the patch passed
+1 💚	golang	4m 6s		the patch passed
+1 💚	javac	4m 6s		the patch passed
+1 💚	compile	4m 1s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	cc	4m 1s		the patch passed
+1 💚	golang	4m 1s		the patch passed
+1 💚	javac	4m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	22m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	88m 12s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		179m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/artifact/out/Dockerfile
GITHUB PR	#4157
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 04de66e00b85 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0cc0e06`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/testReport/
Max. process+thread count	522 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4157/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* The source files for hdfs_tail uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make these tools cross platform.

GauthamBanasandra added 5 commits April 10, 2022 19:37

Make HDFS tail cross platform

f78d88b

Add unit tests

c94b1d1

Add documentation

997f6a4

Remove redundant file

651a3aa

Fix typo

0cc0e06

GauthamBanasandra marked this pull request as ready for review April 11, 2022 14:57

GauthamBanasandra self-assigned this Apr 11, 2022

GauthamBanasandra requested a review from goiri April 11, 2022 14:57

GauthamBanasandra added the HDFS label Apr 11, 2022

goiri approved these changes Apr 11, 2022

View reviewed changes

GauthamBanasandra merged commit e89fd96 into apache:trunk Apr 12, 2022

GauthamBanasandra deleted the tail-x-platform branch April 12, 2022 15:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16474. Make HDFS tail tool cross platform #4157

HDFS-16474. Make HDFS tail tool cross platform #4157

Uh oh!

GauthamBanasandra commented Apr 10, 2022 •

edited

Loading

Uh oh!

hadoop-yetus commented Apr 10, 2022

Uh oh!

hadoop-yetus commented Apr 10, 2022

Uh oh!

hadoop-yetus commented Apr 11, 2022

Uh oh!

hadoop-yetus commented Apr 11, 2022

Uh oh!

Uh oh!

HDFS-16474. Make HDFS tail tool cross platform #4157

HDFS-16474. Make HDFS tail tool cross platform #4157

Uh oh!

Conversation

GauthamBanasandra commented Apr 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

hdfs_tail

For code changes:

Uh oh!

hadoop-yetus commented Apr 10, 2022

Uh oh!

hadoop-yetus commented Apr 10, 2022

Uh oh!

hadoop-yetus commented Apr 11, 2022

Uh oh!

hadoop-yetus commented Apr 11, 2022

Uh oh!

Uh oh!

GauthamBanasandra commented Apr 10, 2022 •

edited

Loading