HBASE-28906 Run nightly tests with multiple Hadoop 3 versions #6356

stoty · 2024-10-08T13:22:28Z

The nightly tests won't run on this PR.

See https://ci-hbase.apache.org/job/Test%20script%20for%20nightly%20script/job/HBASE-28846/
for the hacked-up test PR which tests the contents of this change

ndimiduk · 2024-10-14T11:19:48Z

dev-support/hbase-personality.sh

@@ -589,6 +609,7 @@ function hadoopcheck_rebuild
  fi

  if [[ "${PATCH_BRANCH}" = *"branch-2.5"* ]]; then
+    # TODO remove this on non 2.5 branches ?
    yetus_info "Setting Hadoop 3 versions to test based on branch-2.5 rules"
    if [[ "${QUICK_HADOOPCHECK}" == "true" ]]; then
      hbase_hadoop3_versions="3.2.4 3.3.6 3.4.0"


Should we keep these version number lists managed in a single place? I guess that would be in the Jenkinsfile(s) (sob).

ndimiduk · 2024-10-14T11:21:57Z

dev-support/hbase-personality.sh

+    if [ -n "${TEST_PROFILE}" ]; then
+      extra="${extra} -P${TEST_PROFILE}"
+    else
+      extra="${extra} -PrunAllTests"


nit: fold "runAllTests" in to a default value for this variable at declaration time and then you can set extra unconditionally.

The parameter parsing is already cryptic (for a bash hater like me), adding a default value there would be even more cryptic IMO (and disturb the current relatively easy-to read option processing pattern) .

ndimiduk · 2024-10-14T11:22:51Z

dev-support/Jenkinsfile

+              for (hadoop3_version in hadoop3_versions) {
+                env.HADOOP3_VERSION = hadoop3_version;
+                echo "env.HADOOP3_VERSION" + env.hadoop3_version;
+                stage ('Hadoop 3 cache inner stage') {


Generating stages from a script. Ouch. Yeah, okay.

stoty · 2024-10-14T12:41:13Z

I agree that this is not very nice, but this was least invasive solution I could think of.

IF we were to stay on Jenkins, then it would be worth either converting all Hadoop tests to a single monster Matrix stage, or the whole Jenkinesfile to a fully scripted pipeline, but hopefully this will only be needed for a few months, and we can spend the effort to convert to GH actions instead.

ndimiduk · 2024-10-14T13:23:34Z

I found the declarative version of these files to be more readable and maintainable than the scripted versions...

Don't hold your breath on a GA implementation anytime soon. Infra hasn't acknowledged my ticket yet.

Apache9 · 2024-10-14T15:14:34Z

dev-support/hbase-personality.sh

    if [[ "${QUICK_HADOOPCHECK}" == "true" ]]; then
-      hbase_hadoop3_versions="3.3.6 3.4.0"


You can see the pattern, actually we will use the latest one for a minor release line for quick hadoop check...

stoty · 2024-10-14T17:07:13Z

I agree, defining the Hadoop versions both in the personality file and the Jenkinsfile is far from ideal.

We could remove those from the personality, and pass a parameter from the Jenkinsfile, but then we'd have to define them both in the nigthly and precommit jenkinsfiles, so we'd have to maintain two copies either way.

Or we could have some kind of include file that is ready by both Jenkins and Yetus. That way it would be in one place, but we'd have to add additional logic to parse that in multiple places.

What is your planned release timeline, @Apache9 ? Should we spend some more weeks on this ? The turnaround time for testing these changes is horribly long, basically one run / day.

Apache9 · 2024-10-17T14:09:33Z

Have you seen my last review comment? In general this is a workable solution. The only thing is about the hadoop quick check version.

We have already build with 3.3.5 by default, so for hadoop quick check, we'd better test the newest one instead of testing 3.3.5 again...

Apache-HBase · 2024-10-18T08:06:30Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6356/4/console in case of problems.

Apache-HBase · 2024-10-18T08:06:36Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6356/4/console in case of problems.

Apache-HBase · 2024-10-18T08:07:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 5s		Maven dependency ordering for branch
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 5s		Maven dependency ordering for patch
			_ Other Tests _
		1m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6356/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6356
Optional Tests
uname	Linux b4fa4a748ae4 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `245bea4`
Max. process+thread count	35 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6356/4/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-10-18T08:08:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
+0 🆗	mvndep	0m 5s		Maven dependency ordering for branch
+1 💚	spotless	0m 40s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 5s		Maven dependency ordering for patch
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	spotless	0m 39s		patch has no errors when running spotless:check.
			_ Other Tests _
+0 🆗	asflicense	0m 0s		ASF License check generated no output?
		2m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6356/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6356
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless
uname	Linux 017af88a8f70 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `245bea4`
Max. process+thread count	47 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6356/4/console
versions	git=2.34.1 maven=3.9.8 shellcheck=0.8.0
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2024-10-18T08:11:25Z

Merged to master.

I'm goint to wait for at least one nightly run before beginning the backports.

…#6356) includes HBASE-28929 Set hadoop-three.version in Hadoop 3 backwards compatibility tests includes addendum: workaround for MAPREDUCE-7492 Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> (cherry picked from commit 4be7e39)

…#6399) includes HBASE-28929 Set hadoop-three.version in Hadoop 3 backwards compatibility tests includes addendum: workaround for MAPREDUCE-7492 Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 4be7e39)

…#6356) (apache#6399) includes HBASE-28929 Set hadoop-three.version in Hadoop 3 backwards compatibility tests includes addendum: workaround for MAPREDUCE-7492 includes HBASE-28940 Do not run the backwards compatibility tests with the default Hadoop3 version Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 4be7e39)

stoty requested review from Apache9 and ndimiduk October 8, 2024 13:22

This comment has been minimized.

Sign in to view

ndimiduk approved these changes Oct 14, 2024

View reviewed changes

Apache9 reviewed Oct 14, 2024

View reviewed changes

stoty added 3 commits October 18, 2024 06:51

HBASE-28906 Run nightly tests with multiple Hadoop 3 versions

583a668

revert default Hadoop 3 version to 3.3.5

852feba

revert hbase_hadoop3_versions for QUCIK_HADOOPCHECK

f428435

stoty force-pushed the HBASE-28906 branch from 832c3ab to f428435 Compare October 18, 2024 04:56

This comment has been minimized.

Sign in to view

revert nonsensical globbed mkdir

245bea4

stoty merged commit 4be7e39 into apache:master Oct 18, 2024
1 check passed

		if [[ "${QUICK_HADOOPCHECK}" == "true" ]]; then
		hbase_hadoop3_versions="3.3.6 3.4.0"

HBASE-28906 Run nightly tests with multiple Hadoop 3 versions #6356

HBASE-28906 Run nightly tests with multiple Hadoop 3 versions #6356

Uh oh!

Conversation

stoty commented Oct 8, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimiduk Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

ndimiduk Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

stoty Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

ndimiduk Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

stoty commented Oct 14, 2024

Uh oh!

ndimiduk commented Oct 14, 2024

Uh oh!

Apache9 Oct 14, 2024

Choose a reason for hiding this comment

Uh oh!

stoty Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

stoty commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache9 commented Oct 17, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Oct 18, 2024

Uh oh!

Apache-HBase commented Oct 18, 2024

Uh oh!

Apache-HBase commented Oct 18, 2024

Uh oh!

Apache-HBase commented Oct 18, 2024

Uh oh!

Uh oh!

stoty commented Oct 18, 2024

Uh oh!

Uh oh!

stoty commented Oct 14, 2024 •

edited

Loading