HADOOP-19663. [Part 1] Add RISC-V CRC32 build scaffolding and placeholder (no functional change) #7903
+41
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
PeterPtroc
force-pushed
the
HADOOP-19663
branch
from
1b159b6 to
3d607d4
Compare
|
💔 -1 overall
This message was automatically generated. |
|
Due to some CI infrastructure issues, I will paste the result of validating this patch on a RISC-V machine. Below are the command and the results. Command： Results |
|
Hi @pan3793 @slfan1989 , could you please take a look when you have a moment? This PR adds RISC-V CRC32 scaffolding and keeps behavior unchanged. Happy to address any feedback. Thanks! |
|
@PeterPtroc I suppose most developers here do not have RISC-V env, is it possible to have a docs about how to verify it by leveraging QEMU or some common tools? |
|
@pan3793 Thanks for the suggestion! Below is a concise doc to verify the correctness of the crc32riscv implementation: I mainly verify on RISC‑V by using QEMU together with the openEuler RISC‑V image. Download the image For me, from the above link, download these four files: RISCV_VIRT_CODE.fd, RISCV_VIRT_VARS.fd, openEuler-25.03-riscv64.qcow2.xz, and start_vm.sh; then log in as root with the password: openEuler12#$. Install required packages yum install -y gcc gcc-c++ gcc-gfortran libgcc cmake
yum install -y wget openssl openssl-devel zlib zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel doxygen clang cyrus-sasl cyrus-sasl-devel libtirpc libtirpc-devel
yum install -y java-17-openjdk.riscv64 java-17-openjdk-devel.riscv64 java-17-openjdk-headless.riscv64Install Protobuf 2.5.0 (with RISC‑V patches) mkdir protobuf && cd protobuf
# Fetch sources
git clone https://gitee.com/src-openeuler/protobuf2.git
cd protobuf2
tar -xjf protobuf-2.5.0.tar.bz2
cp *.patch protobuf-2.5.0 && cd protobuf-2.5.0
# Apply patches (adds riscv64 support and build fixes)
patch -p1 < 0001-Add-generic-GCC-support-for-atomic-operations.patch
patch -p1 < protobuf-2.5.0-gtest.patch
patch -p1 < protobuf-2.5.0-java-fixes.patch
patch -p1 < protobuf-2.5.0-makefile.patch
patch -p1 < add-riscv64-support.patch
# Autotools setup
libtoolize
yum install -y automake
automake-1.17 -a
chmod +x configure
# Configure, build, install
./configure --build=riscv64-unknown-linux --prefix=/usr/local/protobuf-2.5.0
make
make check
make install
ldconfig
# Publish protoc 2.5.0 into local Maven repo (riscv64 classifier)
mvn install:install-file \
-DgroupId=com.google.protobuf \
-DartifactId=protoc \
-Dversion=2.5.0 \
-Dclassifier=linux-riscv64 \
-Dpackaging=exe \
-Dfile=/usr/local/protobuf-2.5.0/bin/protoc
cd ..Install Protobuf 3.25.5 # Download and unpack
wget -c https://github.com/protocolbuffers/protobuf/releases/download/v25.5/protobuf-25.5.tar.gz
tar -xzf protobuf-25.5.tar.gz
cd protobuf-25.5
# Abseil dependency
git clone https://github.com/abseil/abseil-cpp third_party/abseil-cpp
# Configure and build
cmake ./ \
-DCMAKE_BUILD_TYPE=RELEASE \
-Dprotobuf_BUILD_TESTS=off \
-DCMAKE_CXX_STANDARD=20 \
-DCMAKE_INSTALL_PREFIX=/usr/local/protobuf-3.25.5
make install -j "$(nproc)"
# Publish protoc 3.25.5 into local Maven repo (riscv64 classifier)
mvn install:install-file \
-DgroupId=com.google.protobuf \
-DartifactId=protoc \
-Dversion=3.25.5 \
-Dclassifier=linux-riscv64 \
-Dpackaging=exe \
-Dfile=/usr/local/protobuf-3.25.5/bin/protoc
# Make protoc available on PATH and verify
sudo ln -sfn /usr/local/protobuf-3.25.5/bin/protoc /usr/local/bin/protoc
protoc --versionVerify CRC32 using Hadoop native # Clone Hadoop
git clone https://github.com/apache/hadoop.git
cd hadoop
# Increase Maven memory
export MAVEN_OPTS="-Xmx8g -Xms6g"
# Build Hadoop Common (native enabled)
nohup mvn -pl hadoop-common-project/hadoop-common -am -Pnative -DskipTests clean install > build.log 2>&1 &
# Point to built native library directory
cd hadoop-common-project/hadoop-common
export HADOOP_COMMON_LIB_NATIVE_DIR="$PWD/target/native/target/usr/local/lib"
export LD_LIBRARY_PATH="$HADOOP_COMMON_LIB_NATIVE_DIR:$LD_LIBRARY_PATH"
# Run the CRC32 native test
nohup mvn -Pnative -Dtest=org.apache.hadoop.util.TestNativeCrc32 \
-Djava.library.path="$HADOOP_COMMON_LIB_NATIVE_DIR" test > test.log 2>&1 & |
|
Hi @cnauroth , could you please have a look? This PR adds RISC-V CRC32 scaffolding and keeps behavior unchanged. Thanks! |
pan3793
approved these changes
Sep 2, 2025
There was a problem hiding this comment.
I created a riscv hadoop dev container(see #7924) and verifed this PR by running
mvn -Pnative -pl :hadoop-common clean install -DskipTests -am
mvn -Pnative -pl :hadoop-common test -Dtest=org.apache.hadoop.util.TestNativeCrc32
got the same results both with and without this patch
|
Hi @brumi1024 , this PR has been open for a while. Could you please take a look when you have time? Thanks! |
|
@steveloughran |
steveloughran
approved these changes
Oct 8, 2025
...p-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_riscv.c
Outdated
Show resolved
Hide resolved
Co-authored-by: gong-flying <gongxiaofei24@iscas.ac.cn>
PeterPtroc
force-pushed
the
HADOOP-19663
branch
from
3d607d4 to
d4a02dc
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
It seems the CI failure (unable to create new native thread) is due to a resource issue on the build agent, not related to the code changes. |
|
Merged
|
|
Thanks @pan3793 @steveloughran for the review and the merge! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Add a RISC-V-specific compilation unit: org/apache/hadoop/util/bulk_crc32_riscv.c.
This PR establishes the foundational build infrastructure for future RISC-V Zbc (CLMUL) CRC32/CRC32C acceleration without changing current behavior. Follow-ups (HADOOP-19655) will introduce HW-accelerated implementations and runtime dispatch.
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?