HADOOP-18646 update Netty dependency #5435

nao-it · 2023-02-26T07:14:22Z

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2023-02-26T08:42:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 4s		trunk passed
+1 💚	compile	0m 23s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	0m 23s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 31s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	61m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s		the patch passed
+1 💚	compile	0m 14s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	0m 14s		the patch passed
+1 💚	compile	0m 14s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	0m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	0m 15s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	21m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 18s		hadoop-project in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		87m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/1/artifact/out/Dockerfile
GITHUB PR	#5435
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 98214de68199 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f852b7c`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/1/testReport/
Max. process+thread count	699 (vs. ulimit of 5500)
modules	C: hadoop-project U: hadoop-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2023-02-27T10:20:51Z

have you updated the LICENSE-binary file as that was undermaintained and actually blocked last week's release?

nao-it · 2023-02-27T10:54:51Z

have you updated the LICENSE-binary file as that was undermaintained and actually blocked last week's release?

Sorry, I didn't know that it needed to be updated additionally. Now I have added the necessary changes to LICENSE-binary

steveloughran · 2023-02-27T11:06:58Z

Right,

I have just done the x86 RC this weekend and I am doing the arm64 one right now, and with a goal of putting the RC2 out for a vote buy about 17:00 UTC.

Is the CVE something to which Hadoop is actually vulnerable to?

Because we have lots of other issues and trying to keep every single transient jar up to date is a losing battle. If I hold off it will cost time and then something else will come up and I absolutely want to get this up for a vote by tomorrow. Also, last minute JAR updates are incredibly dangerous nobody will have any time to have tested the release for regressions. I am scared of them.

I want to get this release out the way and then we can start worrying about what we do in a follow up in a few months time -which can absolutely take this update as it gives us the time to make sure this update works.

So, please make the case for why this CVE should force the cancelling of the in-progress RC. Otherwise given all the other pressing issues we have to fix in this release I really want to say no.

nao-it · 2023-02-27T11:11:52Z

Right,

I have just done the x86 RC this weekend and I am doing the arm64 one right now, and with a goal of putting the RC2 out for a vote buy about 17:00 UTC.

Is the CVE something to which Hadoop is actually vulnerable to?

Because we have lots of other issues and trying to keep every single transient jar up to date is a losing battle. If I hold off it will cost time and then something else will come up and I absolutely want to get this up for a vote by tomorrow. Also, last minute JAR updates are incredibly dangerous nobody will have any time to have tested the release for regressions. I am scared of them.

I want to get this release out the way and then we can start worrying about what we do in a follow up in a few months time -which can absolutely take this update as it gives us the time to make sure this update works.

So, please make the case for why this CVE should force the cancelling of the in-progress RC. Otherwise given all the other pressing issues we have to fix in this release I really want to say no.

I don't have to jump on the outgoing train, you can put a fix in the next release, since the RC for the current one is already available.

steveloughran · 2023-02-27T13:15:43Z

thanks -I'll put it into trunk & 3.3

hadoop-yetus · 2023-02-28T03:25:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 22s		trunk passed
+1 💚	compile	23m 6s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 32s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	25m 34s		trunk passed
-1 ❌	javadoc	4m 36s	/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	7m 21s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	31m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 7s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 15s		the patch passed
+1 💚	compile	22m 38s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 38s		the patch passed
+1 💚	compile	20m 36s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
-1 ❌	javac	20m 36s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt	root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~~20.04-b08 with JDK Private Build-1.8.0_352-8u352-ga-1~~20.04-b08 generated 2 new + 2624 unchanged - 2 fixed = 2626 total (was 2626)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	19m 48s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
-1 ❌	javadoc	4m 24s	/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	root in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	7m 23s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	32m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	724m 45s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 40s		The patch does not generate ASF License warnings.
		991m 56s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.server.namenode.ha.TestObserverNode
	hadoop.mapred.TestShuffleHandler
	hadoop.mapreduce.v2.app.TestRuntimeEstimators

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/2/artifact/out/Dockerfile
GITHUB PR	#5435
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 3a0ea7f6dfcb 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7aa77e8`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/2/testReport/
Max. process+thread count	3641 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2023-03-07T11:21:05Z

all the test failures are unrelated; we have jiras on these being flaky/brittle to timing issues.

could you rebase as there's now merge problems with the license file...this will trigger a new run and we can see if the failures go away

# Conflicts: # LICENSE-binary

nao-it · 2023-03-08T15:32:27Z

all the test failures are unrelated; we have jiras on these being flaky/brittle to timing issues.

could you rebase as there's now merge problems with the license file...this will trigger a new run and we can see if the failures go away

Done

hadoop-yetus · 2023-03-09T06:08:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 13s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 40s		trunk passed
+1 💚	compile	23m 2s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	20m 42s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	mvnsite	25m 17s		trunk passed
+1 💚	javadoc	8m 9s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 22s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	34m 55s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	35m 15s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 1s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 7s		the patch passed
+1 💚	compile	22m 45s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	22m 45s		the patch passed
+1 💚	compile	20m 29s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
-1 ❌	javac	20m 29s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt	root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~~20.04.1-b09 with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~~20.04.1-b09 generated 4 new + 2623 unchanged - 1 fixed = 2627 total (was 2624)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	19m 49s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	7m 47s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 21s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	35m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	724m 48s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 27s		The patch does not generate ASF License warnings.
		996m 48s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.mapreduce.v2.app.TestRuntimeEstimators

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/3/artifact/out/Dockerfile
GITHUB PR	#5435
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 87609d29b95f 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b68c8df`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/3/testReport/
Max. process+thread count	3222 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5435/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM
+1

steveloughran · 2023-03-10T15:29:02Z

merged and credited the author of the jira.

@nao-it if you aren't Aleksandr Nikolaev, tell me and we can revert/resubmit with whichever name you want to use. thanks

degant · 2023-03-16T19:28:11Z

@steveloughran Will this be part of 3.3.5 or a later version?

…ache#5435) This fixes CVE-2022-41881. This also upgrades io.opencensus dependencies to 0.12.3 Contributed by Aleksandr Nikolaev

…ache#5435) This fixes CVE-2022-41881. This also upgrades io.opencensus dependencies to 0.12.3 Contributed by Aleksandr Nikolaev (cherry picked from commit 734f7ab) Conflicts: hadoop-project/pom.xml Change-Id: I26b8961725706370ac5f0fa248d0b0333034a047

) (#5729) This fixes CVE-2022-41881. This also upgrades io.opencensus dependencies to 0.12.3 Contributed by Aleksandr Nikolaev (cherry picked from commit 734f7ab) Conflicts: hadoop-project/pom.xml Change-Id: I26b8961725706370ac5f0fa248d0b0333034a047 Co-authored-by: nao <56360298+nao-it@users.noreply.github.com>

HADOOP-18646 update Netty dependency

f852b7c

fix LICENSE-binary

7aa77e8

Merge branch 'trunk' into HADOOP-18646

b68c8df

# Conflicts: # LICENSE-binary

steveloughran approved these changes Mar 10, 2023

View reviewed changes

steveloughran merged commit 734f7ab into apache:trunk Mar 10, 2023

degant mentioned this pull request Mar 16, 2023

HADOOP-18578. Bump netty to the latest 4.1.86 #5229

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18646 update Netty dependency #5435

HADOOP-18646 update Netty dependency #5435

Uh oh!

nao-it commented Feb 26, 2023 •

edited by steveloughran

Loading

Uh oh!

hadoop-yetus commented Feb 26, 2023

Uh oh!

steveloughran commented Feb 27, 2023

Uh oh!

nao-it commented Feb 27, 2023

Uh oh!

steveloughran commented Feb 27, 2023

Uh oh!

nao-it commented Feb 27, 2023

Uh oh!

steveloughran commented Feb 27, 2023

Uh oh!

hadoop-yetus commented Feb 28, 2023

Uh oh!

steveloughran commented Mar 7, 2023

Uh oh!

nao-it commented Mar 8, 2023

Uh oh!

hadoop-yetus commented Mar 9, 2023

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran commented Mar 10, 2023

Uh oh!

degant commented Mar 16, 2023

Uh oh!

Uh oh!

HADOOP-18646 update Netty dependency #5435

HADOOP-18646 update Netty dependency #5435

Uh oh!

Conversation

nao-it commented Feb 26, 2023 • edited by steveloughran Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Feb 26, 2023

Uh oh!

steveloughran commented Feb 27, 2023

Uh oh!

nao-it commented Feb 27, 2023

Uh oh!

steveloughran commented Feb 27, 2023

Uh oh!

nao-it commented Feb 27, 2023

Uh oh!

steveloughran commented Feb 27, 2023

Uh oh!

hadoop-yetus commented Feb 28, 2023

Uh oh!

steveloughran commented Mar 7, 2023

Uh oh!

nao-it commented Mar 8, 2023

Uh oh!

hadoop-yetus commented Mar 9, 2023

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Mar 10, 2023

Uh oh!

degant commented Mar 16, 2023

Uh oh!

Uh oh!

nao-it commented Feb 26, 2023 •

edited by steveloughran

Loading