HADOOP-19107. Drop support for HBase v1 timeline service & upgrade HBase v2 (#6629).

[steveloughran]

Description of PR

HADOOP-19107 cherrypicked to branch 3.4, targeting 3.4.2 release

the main merge pain was in hadoop-yarn-project/pom.xml ; there's enough of what appears to be unrelated to say "someone who understands yarn modules needs to review this"

How was this patch tested?

it compiles

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	pathlen	0m 0s	/results-pathlen.txt	The patch appears to contain 7 files with names longer than 240
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	4m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	42m 25s		branch-3.4 passed
+1 💚	compile	19m 54s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	18m 41s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	4m 30s		branch-3.4 passed
+1 💚	mvnsite	25m 52s		branch-3.4 passed
+1 💚	javadoc	9m 1s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	8m 17s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 18s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 18s		branch/hadoop-assemblies no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	34m 36s	/branch-spotbugs-root-warnings.html	root in branch-3.4 has 4 extant spotbugs warnings.
-1 ❌	spotbugs	12m 5s	/branch-spotbugs-hadoop-yarn-project-warnings.html	hadoop-yarn-project in branch-3.4 has 1 extant spotbugs warnings.
+1 💚	shadedclient	34m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 36s		Maven dependency ordering for patch
-1 ❌	mvninstall	30m 19s	/patch-mvninstall-root.txt	root in the patch failed.
-1 ❌	mvninstall	2m 55s	/patch-mvninstall-hadoop-yarn-project.txt	hadoop-yarn-project in the patch failed.
-1 ❌	mvninstall	0m 17s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	compile	18m 43s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	18m 43s		the patch passed
+1 💚	compile	17m 54s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	17m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 12s	/buildtool-patch-checkstyle-root.txt	The patch fails to run checkstyle in root
-1 ❌	mvnsite	4m 52s	/patch-mvnsite-root.txt	root in the patch failed.
+1 💚	shellcheck	0m 0s		No new issues.
-1 ❌	javadoc	7m 9s	/patch-javadoc-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	root in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌	javadoc	6m 48s	/patch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt	root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+0 🆗	spotbugs	0m 20s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 21s		hadoop-assemblies has no data from spotbugs
-1 ❌	spotbugs	27m 47s	/patch-spotbugs-root.txt	root in the patch failed.
-1 ❌	spotbugs	11m 42s	/patch-spotbugs-hadoop-yarn-project.txt	hadoop-yarn-project in the patch failed.
-1 ❌	spotbugs	0m 19s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	shadedclient	37m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	759m 57s	/patch-unit-root.txt	root in the patch passed.
-1 ❌	asflicense	1m 43s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		1196m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/1/artifact/out/Dockerfile
GITHUB PR	#7017
Optional Tests	dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle shellcheck shelldocs
uname	Linux 9b4790b3355a 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 9e4cad0
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/1/testReport/
Max. process+thread count	3508 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 . hadoop-yarn-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	pathlen	0m 0s	/results-pathlen.txt	The patch appears to contain 7 files with names longer than 240
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	4m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	43m 11s		branch-3.4 passed
+1 💚	compile	19m 47s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	17m 51s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	4m 21s		branch-3.4 passed
+1 💚	mvnsite	22m 57s		branch-3.4 passed
+1 💚	javadoc	9m 9s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	8m 18s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 19s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 18s		branch/hadoop-assemblies no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	34m 37s	/branch-spotbugs-root-warnings.html	root in branch-3.4 has 4 extant spotbugs warnings.
-1 ❌	spotbugs	11m 57s	/branch-spotbugs-hadoop-yarn-project-warnings.html	hadoop-yarn-project in branch-3.4 has 1 extant spotbugs warnings.
+1 💚	shadedclient	34m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 38s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 19s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	compile	19m 39s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	19m 39s		the patch passed
+1 💚	compile	17m 40s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	17m 40s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 34s		the patch passed
+1 💚	mvnsite	23m 48s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 55s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	8m 8s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 18s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 19s		hadoop-assemblies has no data from spotbugs
-1 ❌	spotbugs	0m 23s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	shadedclient	38m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	765m 2s	/patch-unit-root.txt	root in the patch passed.
-1 ❌	asflicense	1m 50s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		1215m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/2/artifact/out/Dockerfile
GITHUB PR	#7017
Optional Tests	dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle shellcheck shelldocs
uname	Linux 8678e969ae50 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 42ba5d1
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/2/testReport/
Max. process+thread count	4169 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 . hadoop-yarn-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

spotbugs unhappy

Thu 29 Aug 2024 09:05:01 PM UTC
cd /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7017@2/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1
/usr/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7017@2/yetus-m2/hadoop-branch-3.4-patch-0 -DskipTests test-compile spotbugs:spotbugs -DskipTests=true
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-readable POM /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7017@2/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/pom.xml: input contained no data @ 
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project  (/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7017@2/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/pom.xml) has 1 error
[ERROR]     Non-readable POM /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7017@2/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/pom.xml: input contained no data

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	pathlen	0m 0s	/results-pathlen.txt	The patch appears to contain 7 files with names longer than 240
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	4m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	41m 16s		branch-3.4 passed
+1 💚	compile	17m 59s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	16m 11s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	4m 19s		branch-3.4 passed
+1 💚	mvnsite	22m 9s		branch-3.4 passed
+1 💚	javadoc	8m 47s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	7m 54s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 20s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 21s		branch/hadoop-assemblies no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	31m 25s	/branch-spotbugs-root-warnings.html	root in branch-3.4 has 4 extant spotbugs warnings.
-1 ❌	spotbugs	10m 43s	/branch-spotbugs-hadoop-yarn-project-warnings.html	hadoop-yarn-project in branch-3.4 has 1 extant spotbugs warnings.
+1 💚	shadedclient	33m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 35s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 20s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	compile	17m 19s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	17m 19s		the patch passed
+1 💚	compile	16m 20s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	16m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 15s		the patch passed
+1 💚	mvnsite	19m 54s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 39s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	7m 52s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 21s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 20s		hadoop-assemblies has no data from spotbugs
-1 ❌	spotbugs	0m 24s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	shadedclient	33m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	753m 31s	/patch-unit-root.txt	root in the patch passed.
-1 ❌	asflicense	1m 45s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		1175m 20s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/3/artifact/out/Dockerfile
GITHUB PR	#7017
Optional Tests	dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle shellcheck shelldocs
uname	Linux 7efb1a1b097e 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 9fad755
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/3/testReport/
Max. process+thread count	3994 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 hadoop-yarn-project . hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[ayushtkn]

the PR has some changes from HADOOP-19019, maybe they came while trying to resolve conflicts, the build is failing for some strange reason

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-readable POM /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7017/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/pom.xml: input contained no data @ 
 @ 

It is crying for HBase-1 POM:
hadoop-yarn-server-timelineservice-hbase-server-1/pom.xml:
earlier it was there in the directory (steveloughran@9fad755?diff=split&w=0#diff-24caacb8ff3a8c363a257174786a5c6d3ff1a88602a29b38bdf2840014338470) but in the latest commit, I deleted this file, not sure why it is going to hbase-server-1 & looking for its pom, need to investigate...

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	pathlen	0m 0s	/results-pathlen.txt	The patch appears to contain 7 files with names longer than 240
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	4m 18s		Maven dependency ordering for branch
+1 💚	mvninstall	41m 57s		branch-3.4 passed
+1 💚	compile	18m 0s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	16m 34s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	4m 19s		branch-3.4 passed
+1 💚	mvnsite	21m 59s		branch-3.4 passed
+1 💚	javadoc	8m 52s		branch-3.4 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	7m 45s		branch-3.4 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 20s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 22s		branch/hadoop-assemblies no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	31m 9s	/branch-spotbugs-root-warnings.html	root in branch-3.4 has 4 extant spotbugs warnings.
-1 ❌	spotbugs	10m 54s	/branch-spotbugs-hadoop-yarn-project-warnings.html	hadoop-yarn-project in branch-3.4 has 1 extant spotbugs warnings.
+1 💚	shadedclient	33m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 35s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 20s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	compile	16m 58s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	16m 58s		the patch passed
+1 💚	compile	16m 21s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	16m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 12s		the patch passed
+1 💚	mvnsite	20m 20s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 34s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	7m 52s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 21s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 20s		hadoop-assemblies has no data from spotbugs
-1 ❌	spotbugs	0m 24s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed.
+1 💚	shadedclient	33m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	750m 8s	/patch-unit-root.txt	root in the patch passed.
-1 ❌	asflicense	1m 44s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		1172m 15s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.protocol.TestBlockListAsLongs

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/4/artifact/out/Dockerfile
GITHUB PR	#7017
Optional Tests	dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle shellcheck shelldocs
uname	Linux 3a53db7e7e9c 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 2fc01e1
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/4/testReport/
Max. process+thread count	3906 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 hadoop-yarn-project . hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7017/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

one of the test failures will be fixed by 1.3.0 third party

Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
 at org.apache.hadoop.thirdparty.protobuf.IterableByteBufferInputStream.read(IterableByteBufferInputStream.java:143)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.read(CodedInputStream.java:2080)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.tryRefillBuffer(CodedInputStream.java:2831)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2777)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawByte(CodedInputStream.java:2859)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64SlowPath(CodedInputStream.java:2648)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64(CodedInputStream.java:2641)
 at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readSInt64(CodedInputStream.java:2497)
 at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:419)
 at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:397)
 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:3349)
 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:3171)
 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:2950)
 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.lambda$blockReport$0(NameNodeRpcServer.java:1652)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:5637)
 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5614)

[ayushtkn]

Hi @steveloughran I tried experimenting on this PR but for some strange reason it is picking up the hbase-server-1 module in the yetus and failing for that.
I created a WIP PR, backported: HADOOP-19019 + this one from scratch & those error don't pop up there
#7023 (comment)

some unrelated test failures here, So I/you can force push the changes from that PR here & then merge it via CLI, since there are two commits being backported & I think it should be safe to backport HADOOP-19019, it was causing conflicts with this one, so I included that as well.

or figure out what is the diff b/w the two PR and push the delta here

[steveloughran]

if you push up a backport PR into CDPD-64482 we can revisit this after. not targeting 3.4.1 there; it could be the last release with hbase 1 and java8...

[zeekling]

I think there is no need to remove support for HBase v1 in Hadoop 3.4, which is a destructive change.

[steveloughran]

I think there is no need to remove support for HBase v1 in Hadoop 3.4, which is a destructive change.

I shall make clear in the JIRA that this only for ATS...hbase itself should be happy. Though as usual, testing is always welcome.

Is that OK?

[steveloughran]

@zeekling is the change acceptable?

Or is it that you consider a branch-3.4 backport the issue

FWIW I'm thinking we may want to do a 3.5 release with compatibility issues in a few months, including:

• this change
• java11+ only
• some s3a changes which require new options for third party s3 stores.

[slfan1989]

@zeekling is the change acceptable?

Or is it that you consider a branch-3.4 backport the issue

FWIW I'm thinking we may want to do a 3.5 release with compatibility issues in a few months, including:

• this change
• java11+ only
• some s3a changes which require new options for third party s3 stores.

@steveloughran

I support this decision +1.

I support the removal of HBase 1.x in the Hadoop 3.4 branch, as the official stable version of HBase no longer includes 1.x. We should move forward. If users need HBase 1.x, they can utilize previous branches (Hadoop 3.x) and make adaptations suitable for their internal production systems.

https://downloads.apache.org/hbase/

1.0 was EOM'd January 2016; 1.1 was EOM'd December 2017; 1.2 was EOM'd June 2019; 1.3 was EOM'd in August 2020; 1.4 was EOM'd in October 2021; 1.x was EOM'd in August 2022

[NihalJain]

While trying to backport https://issues.apache.org/jira/browse/HADOOP-15760, found that we depend on hadoop 2.8.5 for tests, thanks to hbase 1.7.1 dependency for timeline service, both of which bring in commons-collections:commons-collections:3.2.2 transitively.

I am in support of this change, given hbase 1.x was EOL'ed long ago and having this in hadoop makes things a lot tricky to manage.

[steveloughran]

OK, going ahead with this. We need it for ongoing work.