cherrypick hadoop 3.2.4 changes to branch 3.3.2 #4134

steveloughran · 2022-04-04T12:51:34Z

cherrypick all the changes in hadoop 3.2.4 on top of branch 3.3.2, so they are in sync.

This avoids us having a 3.2.x release which is ahead of the 3.3.x release line.

all the relevant patches were in branch-3.3. which is where I cherry picked them from
This excludes the move off log4j, which is yet to go in to 3.3

If we do a release of this, it should be as a separate chain of patches; this is the wrap up PoC

How was this patch tested?

waiting to see what yetus says

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

steveloughran · 2022-04-04T22:11:38Z

ok, the nuances of hdfs xml parsing break one of the new tests

 org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.testReverseXmlWithoutSnapshotDiffSection
 Error Details

Layout version mismatch.  This oiv tool handles layout version -66, but the XML file has <layoutVersion> -67.  Please either re-generate the XML file with the proper layout version, or manually edit the XML file to be usable with this version of the oiv tool.
 Stack Trace

java.io.IOException: Layout version mismatch.  This oiv tool handles layout version -66, but the XML file has <layoutVersion> -67.  Please either re-generate the XML file with the proper layout version, or manually edit the XML file to be usable with this version of the oiv tool.
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.readVersion(OfflineImageReconstructor.java:1699)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1753)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1846)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.testReverseXmlWithoutSnapshotDiffSection(TestOfflineImageViewer.java:1209)

apache#3926) (cherry picked from commit 125e3b6) Change-Id: I03e4f2af17f0e4a8245c9c2c8ea1cb2cb41f777a

(cherry picked from commit 0e74f1e)

…ternal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) (cherry picked from commit 19d90e6)

…pancies for RC preparation (apache#3991) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 697e5d4)

…shutdown twice. Contributed by Wei-Chiu Chuang. (cherry picked from commit e8cb2ae) Change-Id: I9f04082d650628bc1b8b62dacaaf472f8a578742

… Contributed by Benjamin Teke

…the validation API. Contributed by Benjamin Teke Change-Id: I5505e1b8aaa394dfac31dade7aed6013e0279adc

…hile rename (apache#3898). Contributed by lei w. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

(cherry picked from commit d0fa9b5) Conflicts: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java Change-Id: I2bba28c56dd08da315856066b58b1778b67bfb45 Co-authored-by: Gautham B A <gautham.bangalore@gmail.com>

Reviewed-by: tomscut <litao@bigo.sg> (cherry picked from commit 45ce1cc)

… to prevent proxy server from hanging. Contributed by Qi Zhu. (cherry picked from commit a0deda1) Change-Id: I935725ba094d2c35fdc91dd42883bf5b0d506d56

….per.second (apache#3724) Co-authored-by: gf13871 <gf13871@ly.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 046a620) Change-Id: I2cae5d1c27a492d896da5338a92c7a86f88a8b43

…ache#4120) (cherry picked from commit dc4a680) Change-Id: Ia9ad34b5c3c0f767169fc48a1866c04ff73b1093

…pache#4082)

hadoop-yetus · 2022-04-06T03:48:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	9m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 8 new or modified test files.
			_ branch-3.3.2 Compile Tests _
+0 🆗	mvndep	14m 17s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 42s		branch-3.3.2 passed
+1 💚	compile	18m 2s		branch-3.3.2 passed
+1 💚	checkstyle	2m 54s		branch-3.3.2 passed
+1 💚	mvnsite	6m 52s		branch-3.3.2 passed
+1 💚	javadoc	6m 29s		branch-3.3.2 passed
+1 💚	spotbugs	13m 25s		branch-3.3.2 passed
+1 💚	shadedclient	24m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	5m 2s		the patch passed
+1 💚	compile	17m 29s		the patch passed
-1 ❌	cc	17m 29s	/results-compile-cc-root.txt	root generated 26 new + 133 unchanged - 26 fixed = 159 total (was 159)
+1 💚	javac	17m 29s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 17 line(s) with tabs.
-0 ⚠️	checkstyle	2m 48s	/results-checkstyle-root.txt	root: The patch generated 23 new + 762 unchanged - 20 fixed = 785 total (was 782)
+1 💚	mvnsite	6m 49s		the patch passed
-1 ❌	pylint	0m 4s	/results-pylint.txt	The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	6m 18s		the patch passed
+1 💚	spotbugs	14m 28s		the patch passed
+1 💚	shadedclient	25m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 21s		hadoop-common in the patch passed.
+1 💚	unit	206m 9s		hadoop-hdfs in the patch passed.
+1 💚	unit	1m 10s		hadoop-yarn-api in the patch passed.
+1 💚	unit	4m 48s		hadoop-yarn-common in the patch passed.
+1 💚	unit	1m 22s		hadoop-yarn-server-web-proxy in the patch passed.
+1 💚	unit	92m 10s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	unit	10m 55s		hadoop-mapreduce-client-nativetask in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		539m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4134/2/artifact/out/Dockerfile
GITHUB PR	#4134
Optional Tests	dupname asflicense codespell pylint mvnsite markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle xml cc
uname	Linux e02d0a2c541a 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3.2 / `d33ee73`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4134/2/testReport/
Max. process+thread count	2127 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4134/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2 pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-04-06T09:44:36Z

tests are happy.

all files yetus is complaining about are in sync with branch-3.3; changing them here would be the wrong place/time to do it

aajisaka · 2022-04-07T07:59:43Z

I'm okay with the backports, but the target branch needs to be changed. Hadoop 3.3.2 has been released, so the branch-3.3.2 shouldn't accept any updates. Thank you @steveloughran

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>

hadoop-yetus · 2022-04-08T04:02:25Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	9m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.3.2 Compile Tests _
+0 🆗	mvndep	14m 1s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 53s		branch-3.3.2 passed
+1 💚	compile	18m 3s		branch-3.3.2 passed
+1 💚	checkstyle	2m 54s		branch-3.3.2 passed
+1 💚	mvnsite	25m 4s		branch-3.3.2 passed
+1 💚	javadoc	7m 46s		branch-3.3.2 passed
+0 🆗	spotbugs	0m 17s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 17s		branch/hadoop-assemblies no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 19s		branch/hadoop-client-modules/hadoop-client no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 19s		branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 19s		branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 19s		branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 19s		branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 18s		branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	58m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	49m 11s		the patch passed
+1 💚	compile	17m 38s		the patch passed
-1 ❌	cc	17m 38s	/results-compile-cc-root.txt	root generated 15 new + 144 unchanged - 15 fixed = 159 total (was 159)
+1 💚	javac	17m 38s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-tabs.txt	The patch 17 line(s) with tabs.
-0 ⚠️	checkstyle	2m 51s	/results-checkstyle-root.txt	root: The patch generated 23 new + 765 unchanged - 20 fixed = 788 total (was 785)
+1 💚	mvnsite	20m 48s		the patch passed
-1 ❌	pylint	0m 3s	/results-pylint.txt	The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 44s		The patch has no ill-formed XML file.
+1 💚	javadoc	7m 34s		the patch passed
+0 🆗	spotbugs	0m 16s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 17s		hadoop-assemblies has no data from spotbugs
+0 🆗	spotbugs	0m 19s		hadoop-client-modules/hadoop-client has no data from spotbugs
+0 🆗	spotbugs	0m 19s		hadoop-client-modules/hadoop-client-runtime has no data from spotbugs
+0 🆗	spotbugs	0m 19s		hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
+0 🆗	spotbugs	0m 19s		hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs
+0 🆗	spotbugs	0m 18s		hadoop-client-modules/hadoop-client-check-test-invariants has no data from spotbugs
+0 🆗	spotbugs	0m 19s		hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
+1 💚	shadedclient	58m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	699m 52s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 15s		The patch does not generate ASF License warnings.
		1099m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
	hadoop.hdfs.TestFileCreation
	hadoop.yarn.sls.TestReservationSystemInvariants
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.yarn.csi.client.TestCsiClient

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4134/3/artifact/out/Dockerfile
GITHUB PR	#4134
Optional Tests	dupname asflicense codespell pylint mvnsite markdownlint compile javac javadoc mvninstall unit shadedclient xml spotbugs checkstyle cc shellcheck shelldocs
uname	Linux 8cb8de6ecdcd 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3.2 / `0d70c83`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4134/3/testReport/
Max. process+thread count	3137 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-assemblies hadoop-common-project/hadoop-minikdc hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-auth-examples hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project hadoop-client-modules/hadoop-client hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4134/3/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 spotbugs=4.2.2 pylint=2.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-04-08T13:50:32Z

good point. i will create a new one and start the release process properly...this PR just verifies that the patches took

singer-bin and others added 14 commits April 5, 2022 19:39

HDFS-16437 ReverseXML processor doesn't accept XML files without the … (

25c2b33

apache#3926) (cherry picked from commit 125e3b6) Change-Id: I03e4f2af17f0e4a8245c9c2c8ea1cb2cb41f777a

Fix thread safety of EC decoding during concurrent preads (apache#3881)

0c67a5e

(cherry picked from commit 0e74f1e)

HADOOP-18109. Ensure that default permissions of directories under in…

6138a51

…ternal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) (cherry picked from commit 19d90e6)

HADOOP-18125. Utility to identify git commit / Jira fixVersion discre…

a2edcd9

…pancies for RC preparation (apache#3991) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 697e5d4)

HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is …

3c262f4

…shutdown twice. Contributed by Wei-Chiu Chuang. (cherry picked from commit e8cb2ae) Change-Id: I9f04082d650628bc1b8b62dacaaf472f8a578742

YARN-11075. Explicitly declare serialVersionUID in LogMutation class.…

b366165

… Contributed by Benjamin Teke

YARN-11014. YARN incorrectly validates maximum capacity resources on …

a99cf1c

…the validation API. Contributed by Benjamin Teke Change-Id: I5505e1b8aaa394dfac31dade7aed6013e0279adc

HDFS-16428. Source path with storagePolicy cause wrong typeConsumed w…

53b2d7c

…hile rename (apache#3898). Contributed by lei w. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

HDFS-16501. Print the exception when reporting a bad block (apache#4062)

cb81c45

Reviewed-by: tomscut <litao@bigo.sg> (cherry picked from commit 45ce1cc)

YARN-10720. YARN WebAppProxyServlet should support connection timeout…

ed7ee51

… to prevent proxy server from hanging. Contributed by Qi Zhu. (cherry picked from commit a0deda1) Change-Id: I935725ba094d2c35fdc91dd42883bf5b0d506d56

HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes…

fabbeb6

….per.second (apache#3724) Co-authored-by: gf13871 <gf13871@ly.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 046a620) Change-Id: I2cae5d1c27a492d896da5338a92c7a86f88a8b43

MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ (ap…

0ad812e

…ache#4120) (cherry picked from commit dc4a680) Change-Id: Ia9ad34b5c3c0f767169fc48a1866c04ff73b1093

HDFS-16507. [SBN read] Avoid purging edit log which is in progress (a…

d33ee73

…pache#4082)

steveloughran force-pushed the releases/branch-3.3.3 branch from 6d3df3c to d33ee73 Compare April 5, 2022 18:47

apache deleted a comment from hadoop-yetus Apr 6, 2022

HADOOP-18088. Replace log4j 1.x with reload4j. (apache#4052)

0d70c83

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>

steveloughran closed this Apr 19, 2022

steveloughran deleted the releases/branch-3.3.3 branch April 19, 2022 11:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cherrypick hadoop 3.2.4 changes to branch 3.3.2 #4134

cherrypick hadoop 3.2.4 changes to branch 3.3.2 #4134

steveloughran commented Apr 4, 2022

steveloughran commented Apr 4, 2022

hadoop-yetus commented Apr 6, 2022

steveloughran commented Apr 6, 2022

aajisaka commented Apr 7, 2022

hadoop-yetus commented Apr 8, 2022

steveloughran commented Apr 8, 2022

cherrypick hadoop 3.2.4 changes to branch 3.3.2 #4134

cherrypick hadoop 3.2.4 changes to branch 3.3.2 #4134

Conversation

steveloughran commented Apr 4, 2022

How was this patch tested?

For code changes:

steveloughran commented Apr 4, 2022

hadoop-yetus commented Apr 6, 2022

steveloughran commented Apr 6, 2022

aajisaka commented Apr 7, 2022

hadoop-yetus commented Apr 8, 2022

steveloughran commented Apr 8, 2022