HDFS-16285. Make HDFS ownership tools cross platform #3588

GauthamBanasandra · 2021-10-26T16:31:13Z

Description of PR

The source files for hdfs_chown, hdfs_chmod and hdfs_chgrp uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make these tools cross platform.

How was this patch tested?

I ran the above tools on my Hadoop cluster setup locally. The details of testing are as follows -

hdfs_chgrp

Initially -

Changed the group association of a single file from hadoop to yetus group -

Performed the same operation with the recursive option -

Final state of the files viewed through browser -

hdfs_chmod

Initially -

Changed the permissions of a single file to 777 -

Performed the same operation with the recursive option -

Final state of the files viewed through browser -

hdfs_chown

Initially -

Changed the ownership of a single file from hduser to gautham -

Performed the same operation with the recursive option -

Final state of the files viewed through browser -

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

This reverts commit 4d1ca8de63d09f9d18a49cd82415772f55a7396e.

hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-ownership.h

* Moved these two structs to internal folder.

hadoop-yetus · 2021-12-07T14:44:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 36s		trunk passed
+1 💚	compile	3m 1s		trunk passed
+1 💚	mvnsite	0m 24s		trunk passed
+1 💚	shadedclient	61m 46s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	62m 3s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 13s		the patch passed
+1 💚	cc	3m 13s		the patch passed
+1 💚	golang	3m 13s		the patch passed
+1 💚	javac	3m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shadedclient	22m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	100m 56s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		192m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/artifact/out/Dockerfile
GITHUB PR	#3588
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 1a1a7483b4fd 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `31da028`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/testReport/
Max. process+thread count	584 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-07T17:32:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 1s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 40s		trunk passed
+1 💚	compile	3m 15s		trunk passed
+1 💚	mvnsite	0m 35s		trunk passed
+1 💚	shadedclient	50m 56s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	51m 19s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	3m 4s		the patch passed
+1 💚	cc	3m 4s		the patch passed
+1 💚	golang	3m 4s		the patch passed
+1 💚	javac	3m 4s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	shadedclient	22m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	86m 24s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		167m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/artifact/out/Dockerfile
GITHUB PR	#3588
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 7daff2ad841b 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `31da028`
Default Java	Red Hat, Inc.-1.8.0_312-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/testReport/
Max. process+thread count	520 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-07T20:41:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 10s		trunk passed
+1 💚	compile	3m 1s		trunk passed
+1 💚	mvnsite	0m 21s		trunk passed
+1 💚	shadedclient	66m 28s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	66m 46s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	2m 58s		the patch passed
+1 💚	cc	2m 58s		the patch passed
+1 💚	golang	2m 58s		the patch passed
+1 💚	javac	2m 58s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	30m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	84m 55s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		189m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/artifact/out/Dockerfile
GITHUB PR	#3588
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux edec2cc8c146 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `31da028`
Default Java	Debian-11.0.12+7-post-Debian-2deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/testReport/
Max. process+thread count	582 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-07T23:37:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 24s		trunk passed
+1 💚	compile	3m 19s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	3m 21s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	0m 23s		trunk passed
+1 💚	shadedclient	54m 9s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	54m 27s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	3m 21s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	3m 21s		the patch passed
+1 💚	golang	3m 21s		the patch passed
+1 💚	javac	3m 21s		the patch passed
+1 💚	compile	3m 19s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	cc	3m 19s		the patch passed
+1 💚	golang	3m 19s		the patch passed
+1 💚	javac	3m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 15s		the patch passed
+1 💚	shadedclient	22m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	88m 6s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		175m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/artifact/out/Dockerfile
GITHUB PR	#3588
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 8966cb929b59 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `31da028`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/testReport/
Max. process+thread count	579 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/8/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

GauthamBanasandra force-pushed the ch-x-platform branch from 3fcc4bf to d62f528 Compare October 28, 2021 04:41

This comment has been minimized.

Sign in to view

GauthamBanasandra added 9 commits November 20, 2021 00:03

Make hdfs_chown cross platform

21769fe

Add unit test for hdfs_chown

d6e41c9

Fix unit test failure in hdfs_chown

75ea437

Trigger CI

6edee4e

Revert "Trigger CI"

c20294c

This reverts commit 4d1ca8de63d09f9d18a49cd82415772f55a7396e.

Fix unit tests of hdfs_chown

4942c58

Refactor Ownership into a different struct

8c4e3bb

Refactor chgrp

afc4c7c

Add unit tests for hdfs-chgrp

bbed73b

GauthamBanasandra force-pushed the ch-x-platform branch from 1567775 to bbed73b Compare November 19, 2021 18:34

This comment has been minimized.

Sign in to view

Add hdfs_ownership_obj target to hdfs_chgrp

a40dad9

This comment has been minimized.

Sign in to view

GauthamBanasandra added 7 commits December 3, 2021 22:12

Make hdfs_chmod cross platform

1f68889

Fix description for hdfs-chmod

ae589ef

Add unit tests for hdfs-chmod

cdf5e59

Remove redundant obj construction

d862310

Fix hdfs-chmod unit tests

0a36fb0

Fix documentation

67b62f5

Delete unnecessary files

d019924

This comment has been minimized.

Sign in to view

GauthamBanasandra marked this pull request as ready for review December 5, 2021 09:57

GauthamBanasandra requested a review from goiri December 6, 2021 16:45

goiri reviewed Dec 6, 2021

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs-ownership.h Outdated Show resolved Hide resolved

Refactor Ownership and OwnerState

31da028

* Moved these two structs to internal folder.

goiri approved these changes Dec 7, 2021

View reviewed changes

GauthamBanasandra merged commit 6dddbd4 into apache:trunk Dec 9, 2021

GauthamBanasandra deleted the ch-x-platform branch December 9, 2021 04:47

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16285. Make HDFS ownership tools cross platform (apache#3588)

871645a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16285. Make HDFS ownership tools cross platform #3588

HDFS-16285. Make HDFS ownership tools cross platform #3588

GauthamBanasandra commented Oct 26, 2021 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hadoop-yetus commented Dec 7, 2021

hadoop-yetus commented Dec 7, 2021

hadoop-yetus commented Dec 7, 2021

hadoop-yetus commented Dec 7, 2021

HDFS-16285. Make HDFS ownership tools cross platform #3588

HDFS-16285. Make HDFS ownership tools cross platform #3588

Conversation

GauthamBanasandra commented Oct 26, 2021 • edited Loading

Description of PR

How was this patch tested?

hdfs_chgrp

hdfs_chmod

hdfs_chown

For code changes:

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hadoop-yetus commented Dec 7, 2021

hadoop-yetus commented Dec 7, 2021

hadoop-yetus commented Dec 7, 2021

hadoop-yetus commented Dec 7, 2021

GauthamBanasandra commented Oct 26, 2021 •

edited

Loading