HBASE-26645 Upgrade create-release to support both JDK8 and JDK11 #3999

ndimiduk · 2022-01-04T18:56:13Z

update create-release Dockerfile to use jdk11-headless

Requires #3974

dev-support/create-release/hbase-rm/Dockerfile

ndimiduk · 2022-01-04T23:49:04Z

Canceling patch as this is blocked by a bug in javac. See the issue linked from JIRA.

ndimiduk · 2022-04-01T12:22:22Z

@Apache9 's efforts on migrating our JDK8 dependencies out to hbase-thirdparty means that we can now use maven.compiler.release=8, which he also did. Bump the JDK version we use in our create-release automation. Mind taking another look?

ndimiduk · 2022-04-01T12:24:13Z

Or maybe we cannot do this because these tools are used for creating releases from hbase-thirdparty. Is that the case?

Apache9 · 2022-04-01T12:39:23Z

Or maybe we cannot do this because these tools are used for creating releases from hbase-thirdparty. Is that the case?

For hbase-thirdparty we still need to use JDK8. But anyway, we could install both in the docker file, and if it is for hbase we will use JDK11, for others we will use JDK8?

ndimiduk · 2022-04-21T17:51:33Z

Okay, this change just got complicated. Now we include both JDK versions in the docker file (the easy part) and support the user specifying which JDK they want used for the build (the annoyingly involved part).

And I pushed without doing enough testing. The new parsing code has some bugs. Will be back.

Apache9 · 2022-04-28T15:05:03Z

https://issues.apache.org/jira/browse/HBASE-25983

Seems we can not do release with JDK11 because we can not build the javadoc...

ndimiduk · 2022-04-29T09:08:20Z

https://issues.apache.org/jira/browse/HBASE-25983

Seems we can not do release with JDK11 because we can not build the javadoc...

I saw that. This PR can sit until we have more progress, or I can close it for now, if it's in the way. As you like.

apurtell · 2022-04-30T00:45:37Z

We can leave it open I think, assuming HBASE-25983 is going to be fixed in the near term.

add support for multiple jdk versions Include both jdk8 and jdk11 in our docker image. Extend `do-release-docker.sh` to support selecting a jdk version ; doing so required switching from the bash build-in `getopts` to the less featureful command `getopt`, which supports parsing long command options.

Apache-HBase · 2022-11-09T17:51:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 4s	Maven dependency ordering for patch
		_ Other Tests _
		2m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3999
Optional Tests
uname	Linux b26e958212cd 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b1a648f`
Max. process+thread count	33 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-11-09T17:51:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 4s	Maven dependency ordering for patch
		_ Other Tests _
		2m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3999
Optional Tests
uname	Linux 3fe7dd093963 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b1a648f`
Max. process+thread count	37 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-11-09T17:53:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	spotless	0m 54s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s	Maven dependency ordering for patch
+1 💚	hadolint	0m 0s	There were no new hadolint issues.
+1 💚	shellcheck	0m 3s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	spotless	1m 0s	patch has no errors when running spotless:check.
		_ Other Tests _
+0 🆗	asflicense	0m 0s	ASF License check generated no output?
		4m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3999
Optional Tests	dupname asflicense spotless shellcheck shelldocs hadolint
uname	Linux cf342dc61aca 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b1a648f`
Max. process+thread count	44 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/3/console
versions	git=2.34.1 maven=3.8.6 shellcheck=0.8.0 hadolint=Haskell Dockerfile Linter 2.10.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Ubuntu's update{-java}-alternatives requires sudo and globally modifies symlinks. The release script expects to select a java version dynamically. Thus, use jenv to manage java selection.

Apache9 · 2022-11-10T15:31:53Z

For the upcoming 2.5.2 release where we plan to make both hadoop2 and hadoop3 artifacts, I do not think we can use java11 since it can not be used to build artifacts for hadoop2... Of course it is possible to build hadoop3 artifacts with java11, but anyway, we can not specify java11 only from the command line...

Apache-HBase · 2022-11-10T15:39:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 4s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 5s	Maven dependency ordering for patch
		_ Other Tests _
		2m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3999
Optional Tests
uname	Linux bf2c00e33f6a 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `260cd48`
Max. process+thread count	28 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-11-10T15:39:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 17s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 4s	Maven dependency ordering for patch
		_ Other Tests _
		2m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3999
Optional Tests
uname	Linux 988e48af49a5 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `260cd48`
Max. process+thread count	38 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-11-10T15:41:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 4s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 1s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for branch
+1 💚	spotless	1m 2s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s	Maven dependency ordering for patch
+1 💚	hadolint	0m 1s	There were no new hadolint issues.
+1 💚	shellcheck	0m 3s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	spotless	0m 51s	patch has no errors when running spotless:check.
		_ Other Tests _
+0 🆗	asflicense	0m 0s	ASF License check generated no output?
		4m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3999
Optional Tests	dupname asflicense spotless shellcheck shelldocs hadolint
uname	Linux fa405f6dd959 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `260cd48`
Max. process+thread count	43 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-3999/4/console
versions	git=2.34.1 maven=3.8.6 shellcheck=0.8.0 hadolint=Haskell Dockerfile Linter 2.10.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2022-11-10T17:48:30Z

For the upcoming 2.5.2 release where we plan to make both hadoop2 and hadoop3 artifacts, I do not think we can use java11 since it can not be used to build artifacts for hadoop2... Of course it is possible to build hadoop3 artifacts with java11, but anyway, we can not specify java11 only from the command line...

I think you're right. Doing this with create-release will be strange, the way it current considers a single jvm version as "all or nothing". It's also annoying because of how ubuntu manages multiple openjdk package installs... you can't just set JAVA_HOME to get a java version, the java binary appears to be pinned to it's install location. See my latest commit on this branch, where I start working with jenv to enable switching JVMs more dynamically.

How do you want to proceed for 2.5.2?

ndimiduk · 2022-11-11T15:46:03Z

Should we build two docker images, one with JDK8 and the other with JDK11 installed, have the script launch each build in the appropriate image? The alternative would be to invoke update-java-alternatives (with sudo) between the two builds within the same container image.

Apache9 · 2022-11-11T15:53:31Z

For the upcoming 2.5.2 release where we plan to make both hadoop2 and hadoop3 artifacts, I do not think we can use java11 since it can not be used to build artifacts for hadoop2... Of course it is possible to build hadoop3 artifacts with java11, but anyway, we can not specify java11 only from the command line...

I think you're right. Doing this with create-release will be strange, the way it current considers a single jvm version as "all or nothing". It's also annoying because of how ubuntu manages multiple openjdk package installs... you can't just set JAVA_HOME to get a java version, the java binary appears to be pinned to it's install location. See my latest commit on this branch, where I start working with jenv to enable switching JVMs more dynamically.

How do you want to proceed for 2.5.2?

Just use java8 this time. We still want to keep the java8 compatibility even for hadoop3 so this sohuld not be a problem for now.

And for multi jdks, I think we could follow what we have done in the pre commit docker image, just download Temurin(used to be called Adopt) OpenJDKs for both java8 and java11, add JAVA_HOME/bin to path, and switch JAVA_HOME if we want to use different jdks.

Maybe later, we could use java8 to build hadoop2 related artifacts, and use java11 to build hadoop3 related artifacts. And for the upcoming 3.x releases, since they only support hadoop3, I think we could make them java11 only.

Thanks.

ndimiduk requested review from busbey, saintstack, apurtell, madrob and Apache9 January 4, 2022 18:56

This comment was marked as outdated.

Sign in to view

madrob reviewed Jan 4, 2022

View reviewed changes

dev-support/create-release/hbase-rm/Dockerfile Show resolved Hide resolved

joshelser approved these changes Jan 4, 2022

View reviewed changes

ndimiduk closed this Jan 4, 2022

ndimiduk deleted the 26645-create-release-jdk11 branch January 4, 2022 23:49

ndimiduk restored the 26645-create-release-jdk11 branch January 4, 2022 23:49

ndimiduk reopened this Apr 1, 2022

ndimiduk force-pushed the 26645-create-release-jdk11 branch from 0d0f9bb to 4c592b3 Compare April 1, 2022 12:16

ndimiduk requested review from madrob and joshelser April 1, 2022 12:22

This comment was marked as outdated.

Sign in to view

ndimiduk force-pushed the 26645-create-release-jdk11 branch from deb3a4c to 4e12010 Compare April 21, 2022 17:41

ndimiduk force-pushed the 26645-create-release-jdk11 branch from 4e12010 to 28ad068 Compare April 21, 2022 17:57

This comment was marked as outdated.

Sign in to view

ndimiduk force-pushed the 26645-create-release-jdk11 branch from 28ad068 to aada0c0 Compare November 9, 2022 16:38

ndimiduk changed the title ~~HBASE-26645 Upgrade create-release to use JDK11~~ HBASE-26645 Upgrade create-release to support both JDK8 and JDK11 Nov 9, 2022

ndimiduk force-pushed the 26645-create-release-jdk11 branch from aada0c0 to 9d4a07e Compare November 9, 2022 16:39

use ubuntu:22:04 everywhere and use jenv for JDK selection

73d5a2f

Ubuntu's update{-java}-alternatives requires sudo and globally modifies symlinks. The release script expects to select a java version dynamically. Thus, use jenv to manage java selection.

ndimiduk closed this Jan 18, 2023

HBASE-26645 Upgrade create-release to support both JDK8 and JDK11 #3999

HBASE-26645 Upgrade create-release to support both JDK8 and JDK11 #3999

Uh oh!

Conversation

ndimiduk commented Jan 4, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

ndimiduk commented Jan 4, 2022

Uh oh!

ndimiduk commented Apr 1, 2022

Uh oh!

ndimiduk commented Apr 1, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 commented Apr 1, 2022

Uh oh!

ndimiduk commented Apr 21, 2022

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache9 commented Apr 28, 2022

Uh oh!

ndimiduk commented Apr 29, 2022

Uh oh!

apurtell commented Apr 30, 2022

Uh oh!

Apache-HBase commented Nov 9, 2022

Uh oh!

Apache-HBase commented Nov 9, 2022

Uh oh!

Apache-HBase commented Nov 9, 2022

Uh oh!

Apache9 commented Nov 10, 2022

Uh oh!

Apache-HBase commented Nov 10, 2022

Uh oh!

Apache-HBase commented Nov 10, 2022

Uh oh!

Apache-HBase commented Nov 10, 2022

Uh oh!

ndimiduk commented Nov 10, 2022

Uh oh!

ndimiduk commented Nov 11, 2022

Uh oh!

Apache9 commented Nov 11, 2022

Uh oh!

Uh oh!