Skip to content

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to branch-3 #5418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 21, 2023
Merged

Forward-port "HBASE-28065 Corrupt HFile data is mishandled in several cases" to branch-3 #5418

merged 2 commits into from
Sep 21, 2023

Conversation

ndimiduk
Copy link
Member

No description provided.

@ndimiduk
HBASE-28065 Corrupt HFile data is mishandled in several cases
5c58af1 
* when no block size is provided and there's not a preread headerBuf, treat the value with
  caution.
* verify HBase checksums before making use of the block header.
* inline verifyOnDiskSizeMatchesHeader to keep throw/return logic in the method body.
* separate validation of onDiskSizeWithHeader as input parameter from as read from block header
* simplify branching around fetching and populating onDiskSizeWithHeader.
* inline retrieving nextOnDiskBlockSize ; add basic validation.
* whenever a read is determined to be corrupt and fallback to HDFS checksum is necessary, also
  invalidate the cached value of headerBuf.
* build out a test suite covering various forms of block header corruption, for blocks in first
  and second positions.

Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
@ndimiduk
adjustment for master branch
4cc944f
@ndimiduk ndimiduk added the backport This PR is a back port of some issue or issues already committed to master label Sep 20, 2023
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ branch-3 Compile Tests _
+1 💚 mvninstall 3m 48s branch-3 passed
+1 💚 compile 3m 3s branch-3 passed
+1 💚 checkstyle 0m 43s branch-3 passed
+1 💚 spotless 0m 58s branch has no errors when running spotless:check.
+1 💚 spotbugs 1m 46s branch-3 passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 13s the patch passed
+1 💚 compile 2m 46s the patch passed
+1 💚 javac 2m 46s the patch passed
-0 ⚠️ checkstyle 0m 39s hbase-server: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 hadoopcheck 10m 19s Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚 spotless 0m 40s patch has no errors when running spotless:check.
+1 💚 spotbugs 1m 33s the patch passed
_ Other Tests _
+1 💚 asflicense 0m 10s The patch does not generate ASF License warnings.
36m 31s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #5418
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 10e8bb68ec8c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-3 / 6cac6c2
Default Java Eclipse Adoptium-11.0.17+8
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count 79 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/console
versions git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 20s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-3 Compile Tests _
+1 💚 mvninstall 3m 47s branch-3 passed
+1 💚 compile 1m 3s branch-3 passed
+1 💚 shadedjars 7m 14s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 36s branch-3 passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 50s the patch passed
+1 💚 compile 1m 5s the patch passed
+1 💚 javac 1m 5s the patch passed
+1 💚 shadedjars 6m 10s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 38s the patch passed
_ Other Tests _
-1 ❌ unit 271m 47s hbase-server in the patch failed.
301m 33s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #5418
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8ca36b7d1cb8 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-3 / 6cac6c2
Default Java Eclipse Adoptium-11.0.17+8
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/testReport/
Max. process+thread count 4856 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 43s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-3 Compile Tests _
+1 💚 mvninstall 3m 41s branch-3 passed
+1 💚 compile 1m 1s branch-3 passed
+1 💚 shadedjars 7m 3s branch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 37s branch-3 passed
_ Patch Compile Tests _
+1 💚 mvninstall 3m 11s the patch passed
+1 💚 compile 1m 0s the patch passed
+1 💚 javac 1m 0s the patch passed
+1 💚 shadedjars 7m 13s patch has no errors when building our shaded downstream artifacts.
+1 💚 javadoc 0m 38s the patch passed
_ Other Tests _
+1 💚 unit 304m 33s hbase-server in the patch passed.
334m 37s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR #5418
Optional Tests javac javadoc unit shadedjars compile
uname Linux 132c45082282 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-3 / 6cac6c2
Default Java Temurin-1.8.0_352-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/testReport/
Max. process+thread count 4472 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5418/1/console
versions git=2.34.1 maven=3.8.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk ndimiduk merged commit a4b8c6d into apache:branch-3 Sep 21, 2023
@ndimiduk ndimiduk deleted the 28065-hfile-corrupt-reads-branch-3 branch September 21, 2023 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport This PR is a back port of some issue or issues already committed to master
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ndimiduk @Apache-HBase