Skip to content

HBASE-28951 use unique name of temp recovered edits for each worker #7075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

HBASE-28951 use unique name of temp recovered edits for each worker #7075

wants to merge 3 commits into from
+134 −22

Conversation

Umeshkumar9414
Copy link
Contributor

@Umeshkumar9414 Umeshkumar9414 commented Jun 6, 2025 •
edited
Loading

@Apache-HBase

This comment has been minimized.

Sign in to view
Apache9
Apache9 reviewed Jun 6, 2025
View reviewed changes
Copy link
Contributor

@Apache9 Apache9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this could work. What if 'split to hfile' is enabled?

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractRecoveredEditsOutputSink.java Outdated
@@ -55,9 +55,14 @@ public AbstractRecoveredEditsOutputSink(WALSplitter walSplitter,
/** Returns a writer that wraps a {@link WALProvider.Writer} and its Path. Caller should close. */
protected RecoveredEditsWriter createRecoveredEditsWriter(TableName tableName, byte[] region,
long seqId) throws IOException {
// If multiple worker are splitting a WAL at a same time, both should use unique file name to
// avoid conflict
long workerStartCode =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is startCode enough to make unique directory? Different region server could have same start code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I missed this point, this won't work.
To use the whole RS server name, I was worried about reaching the filename lenght limit. Do you also see this concern on it?

@Apache-HBase

This comment has been minimized.

Sign in to view
@Umeshkumar9414 Umeshkumar9414 requested a review from Apache9 June 16, 2025 21:02
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+1 💚 mvninstall 3m 33s master passed
+1 💚 compile 3m 10s master passed
+1 💚 checkstyle 0m 36s master passed
+1 💚 spotbugs 1m 32s master passed
+1 💚 spotless 0m 45s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 2m 51s the patch passed
+1 💚 compile 3m 12s the patch passed
+1 💚 javac 3m 12s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 35s /results-checkstyle-hbase-server.txt hbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚 spotbugs 1m 35s the patch passed
+1 💚 hadoopcheck 11m 12s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚 spotless 0m 43s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 10s The patch does not generate ASF License warnings.
37m 21s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7075/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #7075
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 2b18f581f729 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 281a44b
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 86 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7075/3/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Umeshkumar9414
Copy link
Contributor Author

I'm not sure if this could work. What if 'split to hfile' is enabled?
OutputSink for 'split to hfile', BoundedRecoveredHFilesOutputSink is using unique (random) name. So in this case we will get hfiles with duplicate content. Somehow if we use same name in both the workers we can escape the problem of duplicate hfiles but still we can't escape from some split failures that leads to retry and delay in splitting.

I created this ticket mainly for RecoveredEditsOutputSink. Can we create another ticket for BoundedRecoveredHFilesOutputSink ?

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 30s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 21s master passed
+1 💚 compile 0m 58s master passed
+1 💚 javadoc 0m 28s master passed
+1 💚 shadedjars 6m 8s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 16s the patch passed
+1 💚 compile 0m 58s the patch passed
+1 💚 javac 0m 58s the patch passed
+1 💚 javadoc 0m 27s the patch passed
+1 💚 shadedjars 6m 3s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 ❌ unit 473m 38s /patch-unit-hbase-server.txt hbase-server in the patch failed.
501m 22s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7075/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #7075
Optional Tests javac javadoc unit compile shadedjars
uname Linux ffed5b0ba7ff 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 281a44b
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7075/3/testReport/
Max. process+thread count 5680 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7075/3/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Umeshkumar9414
Copy link
Contributor Author

@Apache9 , @NihalJain , @apurtell can you please review this PR ? I looked into failed test and found that those are getting timed out on master branch as well, will take some time to figure out what is the issue

@virajjasani virajjasani self-requested a review June 18, 2025 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Apache9 Apache9 Awaiting requested review from Apache9

@virajjasani virajjasani Awaiting requested review from virajjasani

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Umeshkumar9414 @Apache-HBase @Apache9