HBASE-29380 Two concurrent remove peer requests may hang

[Apache9]

No description provided.

[Apache9]

I've successfully reproduced the problem with a simple UT.

Let me think how to fix it properly.

[Apache9]

I refactored the MasterProcedureScheduler so the cleanup work for all type of queues share the same cleanup function, so I think it is enough to just test one type in the UT.

Let's wait for the pre commit result.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 6s		master passed
+1 💚	compile	3m 22s		master passed
+1 💚	checkstyle	0m 36s		master passed
+1 💚	spotbugs	1m 32s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 5s		the patch passed
+1 💚	compile	3m 22s		the patch passed
+1 💚	javac	3m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		hbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	spotbugs	1m 42s		the patch passed
+1 💚	hadoopcheck	11m 58s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 9s		The patch does not generate ASF License warnings.
		39m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7077/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7077
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 1de489362706 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 32d82b0
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7077/2/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 15s		master passed
+1 💚	compile	0m 56s		master passed
+1 💚	javadoc	0m 28s		master passed
+1 💚	shadedjars	5m 58s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 2s		the patch passed
+1 💚	compile	0m 57s		the patch passed
+1 💚	javac	0m 57s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed
+1 💚	shadedjars	6m 3s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	217m 26s		hbase-server in the patch passed.
		243m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7077/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7077
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 4cf092f63ab1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 32d82b0
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7077/2/testReport/
Max. process+thread count	5634 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7077/2/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Copilot]

Pull Request Overview

This PR addresses HBASE-29380 by adding a test that reproduces the hanging scenario for concurrent peer-removal requests and refactors the scheduler’s cleanup logic to unify and extend support to global queues and table deletion.

• Introduce TestProcedureWaitAndWake to simulate two concurrent remove-peer procedures and ensure they complete.
• Consolidate per-entity cleanup (peer, server, global) into a generic tryCleanupQueue helper.
• Add a markTableAsDeleted method for safely removing table queues under test restrictions.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureWaitAndWake.java	Adds a new test to simulate and verify that two concurrent remove-peer procedures do not hang.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java	Refactors cleanup methods into a generic tryCleanupQueue, adds global‐queue cleanup, and exposes markTableAsDeleted for tests.

Comments suppressed due to low confidence (3)

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureWaitAndWake.java:166

• The test calls waitFor but does not assert its result. If the condition times out, the test will still pass. Consider wrapping with an assertTrue on the waitFor return value to ensure failures are detected.

UTIL.waitFor(10000, () -> procExec.isFinished(id1));

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestProcedureWaitAndWake.java:167

• As above, add an assertion for the waitFor return value to ensure the second procedure also finishes within the timeout.

UTIL.waitFor(10000, () -> procExec.isFinished(id2));

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java:608

• [nitpick] The parameter getMap is a Supplier<TNode> but its name is ambiguous. Consider renaming to queueSupplier or mapSupplier for clarity.

private <T extends Comparable<T>, TNode extends Queue<T>> boolean tryCleanupQueue(T id,

[ndimiduk]

Nice find. Nice test.