Skip to content

Conversation

@pan3793
Copy link
Member

@pan3793 pan3793 commented Nov 12, 2025

Backport #7780 to branch-3.4

Reviewed-by: Steve Loughran stevel@apache.org

Note: I verified it again locally in both x86 and arm64 env.

$ ./start-build-env.sh
... (then run in container)
$ mvn clean install -DskipTests -Pnative -DskipShade
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:05 min
[INFO] Finished at: 2025-11-12T02:31:20Z
[INFO] ------------------------------------------------------------------------

…#7780)

* HADOOP-19605. Upgrade Protobuf 3.25.5 for docker images.

Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/1/console in case of problems.

@slfan1989
Copy link
Contributor

@pan3793 This PR is not a clean backport, so I can’t give it a +1.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.4 Compile Tests _
+0 🆗 mvndep 2m 15s Maven dependency ordering for branch
+1 💚 mvninstall 23m 9s branch-3.4 passed
-1 ❌ compile 1m 55s /branch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in branch-3.4 failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ compile 1m 40s /branch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in branch-3.4 failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
+1 💚 mvnsite 14m 31s branch-3.4 passed
+1 💚 shadedclient 18m 49s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for patch
+1 💚 mvninstall 17m 0s the patch passed
+1 💚 compile 8m 51s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
-1 ❌ cc 8m 51s /results-compile-cc-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 generated 164 new + 13 unchanged - 32 fixed = 177 total (was 45)
+1 💚 golang 8m 51s the patch passed
+1 💚 javac 8m 51s the patch passed
+1 💚 compile 8m 10s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
-1 ❌ cc 8m 10s /results-compile-cc-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09 with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09 generated 164 new + 13 unchanged - 32 fixed = 177 total (was 45)
+1 💚 golang 8m 10s the patch passed
+1 💚 javac 8m 10s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 hadolint 0m 4s No new issues.
+1 💚 mvnsite 11m 19s the patch passed
-1 ❌ shellcheck 0m 1s /results-shellcheck.txt The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 18m 48s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 650m 54s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 0m 56s The patch does not generate ASF License warnings.
787m 15s
Reason Tests
Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding
hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
hadoop.security.ssl.TestDelegatingSSLSocketFactory
hadoop.mapred.gridmix.TestSleepJob
hadoop.mapred.gridmix.TestGridmixSubmission
hadoop.mapred.gridmix.TestLoadJob
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/1/artifact/out/Dockerfile
GITHUB PR #8078
Optional Tests dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit compile cc javac golang
uname Linux d95d3b3d5d9b 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / c81fdf9
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/1/testReport/
Max. process+thread count 5081 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/1/console
versions git=2.25.1 maven=3.6.3 hadolint=1.11.1-0-g0e692dd shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/2/console in case of problems.

@pan3793
Copy link
Member Author

pan3793 commented Nov 12, 2025

@slfan1989, there are a few differences in trunk and branch-3.4, but the essential change is same, I manually resolved the conflicts and verified again to ensure the backport works.

Since the Java side protobuf upgrading happens in branch-3.4, we should land this in 3.4 too.

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/3/console in case of problems.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 24s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.4 Compile Tests _
+0 🆗 mvndep 0m 34s Maven dependency ordering for branch
-1 ❌ mvninstall 0m 20s /branch-mvninstall-root.txt root in branch-3.4 failed.
-1 ❌ compile 0m 20s /branch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in branch-3.4 failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ compile 2m 6s /branch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in branch-3.4 failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
-1 ❌ mvnsite 0m 21s /branch-mvnsite-root.txt root in branch-3.4 failed.
+1 💚 shadedclient 0m 40s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 21s /patch-mvninstall-root.txt root in the patch failed.
-1 ❌ mvninstall 0m 22s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
-1 ❌ compile 0m 21s /patch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ cc 0m 21s /patch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ golang 0m 21s /patch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ javac 0m 21s /patch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ compile 0m 20s /patch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
-1 ❌ cc 0m 20s /patch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
-1 ❌ golang 0m 20s /patch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
-1 ❌ javac 0m 20s /patch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in the patch failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 hadolint 0m 5s No new issues.
-1 ❌ mvnsite 0m 20s /patch-mvnsite-root.txt root in the patch failed.
-1 ❌ shellcheck 0m 0s /results-shellcheck.txt The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ shadedclient 0m 58s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 20s /patch-unit-root.txt root in the patch failed.
+0 🆗 asflicense 0m 21s ASF License check generated no output?
10m 20s
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/2/artifact/out/Dockerfile
GITHUB PR #8078
Optional Tests dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit compile cc javac golang
uname Linux 5e889f6ad976 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / 2e898a9
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/2/testReport/
Max. process+thread count 56 (vs. ulimit of 5500)
modules C: . hadoop-hdfs-project/hadoop-hdfs-native-client U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/2/console
versions git=2.25.1 maven=3.6.3 hadolint=1.11.1-0-g0e692dd shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 18m 18s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.4 Compile Tests _
+0 🆗 mvndep 2m 22s Maven dependency ordering for branch
+1 💚 mvninstall 39m 15s branch-3.4 passed
-1 ❌ compile 3m 52s /branch-compile-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root in branch-3.4 failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.
-1 ❌ compile 3m 34s /branch-compile-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root in branch-3.4 failed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.
+1 💚 mvnsite 22m 2s branch-3.4 passed
+1 💚 shadedclient 36m 21s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 34m 50s the patch passed
+1 💚 compile 20m 40s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
-1 ❌ cc 20m 40s /results-compile-cc-root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt root-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 generated 164 new + 13 unchanged - 32 fixed = 177 total (was 45)
+1 💚 golang 20m 40s the patch passed
+1 💚 javac 20m 40s the patch passed
+1 💚 compile 19m 13s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
-1 ❌ cc 19m 13s /results-compile-cc-root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09.txt root-jdkPrivateBuild-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09 with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09 generated 164 new + 13 unchanged - 32 fixed = 177 total (was 45)
+1 💚 golang 19m 13s the patch passed
+1 💚 javac 19m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 hadolint 0m 6s No new issues.
+1 💚 mvnsite 19m 18s the patch passed
-1 ❌ shellcheck 0m 0s /results-shellcheck.txt The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 36m 22s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 682m 16s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 13s The patch does not generate ASF License warnings.
944m 53s
Reason Tests
Unreaped Processes root:2
Failed junit tests hadoop.security.ssl.TestDelegatingSSLSocketFactory
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/3/artifact/out/Dockerfile
GITHUB PR #8078
Optional Tests dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit compile cc javac golang
uname Linux cd96a29c4511 5.15.0-160-generic #170-Ubuntu SMP Wed Oct 1 10:06:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / 2e898a9
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Unreaped Processes Log https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/3/artifact/out/patch-unit-root-reaper.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/3/testReport/
Max. process+thread count 3774 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/3/console
versions git=2.25.1 maven=3.6.3 hadolint=1.11.1-0-g0e692dd shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793
Copy link
Member Author

pan3793 commented Nov 24, 2025

@steveloughran do you want to include this in 3.4.3, to make the protobuf cpp version match Java's one?

Native tests in both hadoop-common and hadoop-hdfs-native-client modules are passed.

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8078/3/artifact/out/patch-unit-root.txt

--- maven-antrun-plugin:1.7:run (common-test-bats-driver) @ hadoop-common ---
...
--- maven-antrun-plugin:1.7:run (native_tests) @ hadoop-hdfs-native-client ---
...

@slfan1989
Copy link
Contributor

-1

According to our previous code merge guidelines, we should keep trunk and branch-3.4 as consistent as possible. From what I can see, this PR introduces quite a large difference. Are all of these changes strictly necessary? Would it be possible to meet the requirements without adding these extra modifications? I’d prefer not to let personal preferences affect our overall standards. Thanks again for your support and cooperation!

@pan3793
Copy link
Member Author

pan3793 commented Nov 24, 2025

@slfan1989 I'm not sure what you mean "quite a large difference", compared to the original patch (#7780), except for comments, the only difference is that this backport does not include dev-support/docker/vcpkg/vcpkg.json because it does not exist in branch-3.4.

The backport decision is based on that HADOOP-19289 goes to branch-3.4 but has not completed the work, it only upgrades the Java side, keeping that branch-3.4 uses two different version of protobuf libs.

I'd like to know the technical justification behind the veto.

@slfan1989
Copy link
Contributor

@pan3793 What I want to convey is that, compared to the trunk branch, we've missed several earlier PRs during the compilation of this part. Many of these PRs were provided by you. Should we consider incorporating these earlier changes before proceeding with further modifications?

The relevant PRs are as follows:

@pan3793
Copy link
Member Author

pan3793 commented Nov 24, 2025

@slfan1989 none of these are required.

IIRC, previously we had a conclusion on how to maintain those Dockerfiles:

  • keep them up to date for trunk
  • for released branches, e.g. branch-3.4, keep Dockerfiles permanently on the source tree as a reference for users who want to build on such old platforms, and disable them from the Jenkins pipeline once it reaches EOL and breaks the CI

In branch-3.4, there are 4 Dockerfiles and all of them in the Jenkins pipeline - ubuntu_focal, centos_7, centos_8, debian_10, I think the right thing here is:

  • disable centos_7 on Jenkins pipeline (broken due to old version of glibc does not work with protobuf used by Hadoop Java, thus breaks the build)
  • disable debian_10 on Jenkins pipeline (broken due to APT repo sunset)
  • ubuntu_focal passes CI
  • centos_8 also keeps working, I tested it manually

BTW, disabling centos_7 and debian_10 from the Jenkins pipeline is orthogonal to this PR because they were broken due to different reasons without a relationship to this one.

@slfan1989
Copy link
Contributor

@pan3793 Thank you for the effort you've put into this PR, but I would prefer a clear merge path and an executable specification, rather than relying on verbal explanations about whether this PR is viable. From my understanding, we need a clean backport. Currently, the differences in the base are significant, and forcing a backport in this manner is hard for me to agree with. I also hope this does not set a precedent for the future.

@slfan1989
Copy link
Contributor

disable debian_10 on Jenkins pipeline (broken due to APT repo sunset)

If you think this is reasonable, why force the upgrade to Debian 11 on the trunk?

centos_8 also keeps working, I tested it manually

As with the previous question, why push for an upgrade to RockLinux 8?

@pan3793
Copy link
Member Author

pan3793 commented Nov 24, 2025

disable debian_10 on Jenkins pipeline (broken due to APT repo sunset)

If you think this is reasonable, why force the upgrade to Debian 11 on the trunk?

centos_8 also keeps working, I tested it manually

As with the previous question, why push for an upgrade to RockLinux 8?

First of all, I suppose upgrading the building OS, e.g., Debian from 10 to 11, is kind of a riskier change than a normal dependency upgrade (e.g., protobuf, jackson), for those changes, trunk should move forward, but feature branches should stay at the cut point. I think this is a common policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants