HADOOP-19219. Add JPMS options required by hadoop-common #7084

pan3793 · 2024-09-30T19:02:40Z

Description of PR

JIRA: HADOOP-19219. Add JPMS options required by hadoop-common.

Adding JPMS (JEP 261: Module System) options that required by JDK 17+ for testing (maven-surefire-plugin) and runtime (hadoop-functions.sh)

This PR is based on #6939.

How was this patch tested?

Tested on x86 Linux Java 17, this PR fixes the following cases:

hadoop-auth

[ERROR] org.apache.hadoop.security.authentication.server.TestLdapAuthenticationHandler  Time elapsed: 1.14 s  <<< ERROR!
java.lang.IllegalAccessError: class org.apache.directory.server.core.security.CertificateUtil (in unnamed module @0x3196eefd) cannot access class sun.security.x509.X500Name (in module java.base) because module java.base does not export sun.security.x509 to unnamed module @0x3196eefd
[ERROR] org.apache.hadoop.security.authentication.server.TestMultiSchemeAuthenticationHandler  Time elapsed: 1.299 s  <<< ERROR!
java.lang.IllegalAccessError: class org.apache.directory.server.core.security.CertificateUtil (in unnamed module @0x54a097cc) cannot access class sun.security.util.ObjectIdentifier (in module java.base) because module java.base does not export sun.security.util to unnamed module @0x54a097cc

hadoop-common

[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.155 s <<< FAILURE! - in org.apache.hadoop.io.compress.TestCodecPool
[ERROR] testCompressorConf(org.apache.hadoop.io.compress.TestCodecPool)  Time elapsed: 0.008 s  <<< ERROR!
java.lang.reflect.InaccessibleObjectException: Unable to make field private int java.util.zip.Deflater.level accessible: module java.base does not "opens java.util.zip" to unnamed module @7ca48474

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-09-30T23:02:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 6s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 59s		trunk passed
+1 💚	compile	17m 24s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	16m 33s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	mvnsite	2m 26s		trunk passed
+1 💚	javadoc	1m 59s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	38m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 14s		the patch passed
+1 💚	compile	17m 18s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	17m 17s		the patch passed
+1 💚	compile	16m 14s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	16m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	2m 23s		the patch passed
+1 💚	shellcheck	0m 4s		No new issues.
+1 💚	javadoc	1m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 34s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	40m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 38s		hadoop-project in the patch passed.
+1 💚	unit	20m 24s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 5s		The patch does not generate ASF License warnings.
		238m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7084/1/artifact/out/Dockerfile
GITHUB PR	#7084
Optional Tests	dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient xmllint
uname	Linux c628261a26ac 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5176c4e`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7084/1/testReport/
Max. process+thread count	3152 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7084/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2024-10-01T05:56:20Z

LGTM.

hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh

hadoop-project/pom.xml

hadoop-yetus · 2024-10-01T12:12:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	39m 19s		trunk passed
+1 💚	compile	19m 47s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	16m 13s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	mvnsite	2m 23s		trunk passed
+1 💚	javadoc	1m 54s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 30s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	40m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 14s		the patch passed
+1 💚	compile	18m 23s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	18m 23s		the patch passed
+1 💚	compile	18m 7s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	18m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	2m 23s		the patch passed
+1 💚	shellcheck	0m 5s		No new issues.
+1 💚	javadoc	1m 42s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	46m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 32s		hadoop-project in the patch passed.
+1 💚	unit	20m 52s		hadoop-common in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		247m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7084/2/artifact/out/Dockerfile
GITHUB PR	#7084
Optional Tests	dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient xmllint
uname	Linux 19875fac907b 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0dc4289`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7084/2/testReport/
Max. process+thread count	3152 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7084/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2024-10-02T02:47:44Z

@pan3793 Thanks for the contribution! I want to push directly to branch-3.4, but if this PR involves compilation, it's better to submit a separate PR to handle it. If you have time, please also submit a PR for branch-3.4. @steveloughran Thanks for the review!

… Contributed by Cheng Pan. Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>

Contributed by Cheng Pan. Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>

HADOOP-19219. Add JPMS options required by hadoop-common

5176c4e

github-actions bot added build trunk Common labels Sep 30, 2024

pan3793 mentioned this pull request Sep 30, 2024

HADOOP-19219. Resolve Certificate error in Hadoop-auth tests. #6939

Closed

slfan1989 reviewed Oct 1, 2024

View reviewed changes

hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh Outdated Show resolved Hide resolved

fix

0dc4289

slfan1989 approved these changes Oct 1, 2024

View reviewed changes

slfan1989 reviewed Oct 1, 2024

View reviewed changes

hadoop-project/pom.xml Show resolved Hide resolved

steveloughran approved these changes Oct 1, 2024

View reviewed changes

slfan1989 merged commit 3f637ef into apache:trunk Oct 2, 2024
1 of 3 checks passed

pan3793 mentioned this pull request Oct 2, 2024

HADOOP-19219. (3.4) Add JPMS options required by hadoop-common #7091

Merged

slfan1989 pushed a commit that referenced this pull request Oct 12, 2024

HADOOP-19219. Add JPMS options required by hadoop-common (#7084) (#7091)

85b5bd2

Contributed by Cheng Pan. Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19219. Add JPMS options required by hadoop-common #7084

HADOOP-19219. Add JPMS options required by hadoop-common #7084

Uh oh!

pan3793 commented Sep 30, 2024 •

edited by slfan1989

Loading

Uh oh!

hadoop-yetus commented Sep 30, 2024

Uh oh!

slfan1989 commented Oct 1, 2024

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Oct 1, 2024

Uh oh!

Uh oh!

slfan1989 commented Oct 2, 2024

Uh oh!

Uh oh!

HADOOP-19219. Add JPMS options required by hadoop-common #7084

HADOOP-19219. Add JPMS options required by hadoop-common #7084

Uh oh!

Conversation

pan3793 commented Sep 30, 2024 • edited by slfan1989 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Sep 30, 2024

Uh oh!

slfan1989 commented Oct 1, 2024

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Oct 1, 2024

Uh oh!

Uh oh!

slfan1989 commented Oct 2, 2024

Uh oh!

Uh oh!

pan3793 commented Sep 30, 2024 •

edited by slfan1989

Loading