HBASE-29279 Allow throttling alter operation via table configuration #6951

junegunn · 2025-04-30T15:31:04Z

See https://issues.apache.org/jira/browse/HBASE-29279

Examples

# Use global defaults
alter 't', { NAME => 'd', COMPRESSION => 'GZ' }

# Apply custom throttling for this alter operation
alter 't', { NAME => 'd', COMPRESSION => 'GZ' },
      {
        CONFIGURATION => {
          'hbase.reopen.table.regions.progressive.batch.size.max' => 32,
          'hbase.reopen.table.regions.progressive.batch.backoff.ms' => 500
        }
      }

# Future alter operations on this table will reuse the throttling configuration
alter 't', { NAME => 'd', COMPRESSION => 'GZ' }

junegunn · 2025-04-30T15:38:31Z

...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java

  public static final int PROGRESSIVE_BATCH_SIZE_MAX_DISABLED = -1;
-  private static final int PROGRESSIVE_BATCH_SIZE_MAX_DEFAULT_VALUE = Integer.MAX_VALUE;


This one was unused.

junegunn · 2025-05-01T13:53:50Z

Hi @rmdmattingly, when you have a moment, could you take a look at this? It's a simple change but I believe it makes the throttling feature much more useful in production.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java

junegunn · 2025-05-02T02:01:29Z

...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java

  public static final long PROGRESSIVE_BATCH_BACKOFF_MILLIS_DEFAULT = 0L;
  public static final String PROGRESSIVE_BATCH_SIZE_MAX_KEY =
-    "hbase.reopen.table.regions.progressive.batch.size.max";
+    ConfigKey.INT("hbase.reopen.table.regions.progressive.batch.size.max");


Type checks are now necessary.

Add ReopenTableRegionsProcedure.throttled to create a procedure with the throttling configuration.

Apache-HBase · 2025-05-06T08:14:41Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 14s		master passed
+1 💚	compile	3m 16s		master passed
+1 💚	checkstyle	0m 35s		master passed
+1 💚	spotbugs	1m 36s		master passed
+1 💚	spotless	0m 47s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 5s		the patch passed
+1 💚	compile	3m 17s		the patch passed
+1 💚	javac	3m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	spotbugs	1m 40s		the patch passed
+1 💚	hadoopcheck	11m 54s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		38m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6951/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6951
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux e55aefbad18d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3f520a4`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6951/4/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-05-06T11:33:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 16s		master passed
+1 💚	compile	0m 58s		master passed
+1 💚	javadoc	0m 28s		master passed
+1 💚	shadedjars	5m 53s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 7s		the patch passed
+1 💚	compile	0m 57s		the patch passed
+1 💚	javac	0m 57s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed
+1 💚	shadedjars	5m 51s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	211m 33s		hbase-server in the patch passed.
		237m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6951/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6951
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux c047c7f27940 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3f520a4`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6951/4/testReport/
Max. process+thread count	5445 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6951/4/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2025-06-01T14:07:50Z

...rver/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java

+   */
+  public static ReopenTableRegionsProcedure throttled(final Configuration conf,
+    final TableDescriptor desc) {
+    long backoffMillis = Optional.ofNullable(desc.getValue(PROGRESSIVE_BATCH_BACKOFF_MILLIS_KEY))


Pity we do not return Optional for desc.getValue...

…6951)

…pache#6951)

junegunn · 2025-06-13T08:05:00Z

@Apache9 Thanks for the approval. I've opened a PR to backport this to branch-2. But do you think it's okay to backport this to branch-2.6 as well? I personally think it's a great enhancement of the feature, and it's safe to backport because it's completely backward compatible, but I'm curious how you see it.

NihalJain · 2025-06-16T15:09:20Z

Hi @junegunn welcome to hbase committers' group. I noticed you have merged a few PRs and missed to mention signed-off to those. Not a problem, even I did same when I became a new committer.

But this doc helped me understand how-to's specific to Apache HBase, might be useful to you as well. I wish this (HBASE-21701) was part of our official book!

CC: @chandrasekhar-188k @guluo2016 @charlesconnell

junegunn · 2025-06-16T15:30:24Z

Thank you @NihalJain for helping me out, and apologies for missing that part! Could you share your process of adding the "Signed-off-by" and "Reviewed-by" lines to the commit message? Do I have to manually collect the email addresses of the reviewers and write the lines, or is there an easier way to do it?

NihalJain · 2025-06-16T16:25:34Z

Thank you @NihalJain for helping me out, and apologies for missing that part! Could you share your process of adding the "Signed-off-by" and "Reviewed-by" lines to the commit message? Do I have to manually collect the email addresses of the reviewers and write the lines, or is there an easier way to do it?

I think its all manual, I usually refer hbase teams https://hbase.apache.org/team.html page, or old commits to get email id for a user; if still cant find it, I check their github homepage. If neither of these help, I skip adding it.

junegunn · 2025-06-17T02:38:53Z

I see, thanks. I guess it'll get easier once I get used to the process. I can imagine writing up a script to automate as much of it as possible. Could it be a good candidate for a dev-support script? I'll let you know of the progress.

junegunn mentioned this pull request Apr 30, 2025

HBASE-29245 Region reopening batch size should be increased when backoff is 0 #6892

Merged

junegunn commented Apr 30, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

HBASE-29279 Allow throttling alter operation via table configuration

2e9b0df

junegunn force-pushed the HBASE-29279 branch from 3d0590d to 2e9b0df Compare April 30, 2025 23:02

This comment has been minimized.

Sign in to view

junegunn commented May 2, 2025

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java Outdated Show resolved Hide resolved

junegunn commented May 2, 2025

View reviewed changes

junegunn force-pushed the HBASE-29279 branch 3 times, most recently from 31c0c8d to 5529cb4 Compare May 5, 2025 11:33

This comment has been minimized.

Sign in to view

junegunn force-pushed the HBASE-29279 branch from 5529cb4 to 79702ea Compare May 6, 2025 07:33

Refactor the code

3f520a4

Add ReopenTableRegionsProcedure.throttled to create a procedure with the throttling configuration.

junegunn force-pushed the HBASE-29279 branch from 79702ea to 3f520a4 Compare May 6, 2025 07:34

Apache9 approved these changes Jun 1, 2025

View reviewed changes

junegunn merged commit 145972f into apache:master Jun 13, 2025
1 check passed

junegunn added a commit that referenced this pull request Jun 13, 2025

HBASE-29279 Allow throttling alter operation via table configuration (#…

9118dbb

…6951)

junegunn added a commit to junegunn/hbase that referenced this pull request Jun 13, 2025

HBASE-29279 Allow throttling alter operation via table configuration (a…

e589884

…pache#6951)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-29279 Allow throttling alter operation via table configuration #6951

HBASE-29279 Allow throttling alter operation via table configuration #6951

Uh oh!

junegunn commented Apr 30, 2025 •

edited

Loading

Uh oh!

junegunn Apr 30, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

junegunn commented May 1, 2025

Uh oh!

Uh oh!

junegunn May 2, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented May 6, 2025

Uh oh!

Apache-HBase commented May 6, 2025

Uh oh!

Apache9 Jun 1, 2025

Uh oh!

Uh oh!

junegunn commented Jun 13, 2025

Uh oh!

NihalJain commented Jun 16, 2025

Uh oh!

junegunn commented Jun 16, 2025

Uh oh!

NihalJain commented Jun 16, 2025

Uh oh!

junegunn commented Jun 17, 2025

Uh oh!

Uh oh!

		public static final int PROGRESSIVE_BATCH_SIZE_MAX_DISABLED = -1;
		private static final int PROGRESSIVE_BATCH_SIZE_MAX_DEFAULT_VALUE = Integer.MAX_VALUE;

HBASE-29279 Allow throttling alter operation via table configuration #6951

HBASE-29279 Allow throttling alter operation via table configuration #6951

Uh oh!

Conversation

junegunn commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Examples

Uh oh!

junegunn Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

junegunn commented May 1, 2025

Uh oh!

Uh oh!

junegunn May 2, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented May 6, 2025

Uh oh!

Apache-HBase commented May 6, 2025

Uh oh!

Apache9 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

junegunn commented Jun 13, 2025

Uh oh!

NihalJain commented Jun 16, 2025

Uh oh!

junegunn commented Jun 16, 2025

Uh oh!

NihalJain commented Jun 16, 2025

Uh oh!

junegunn commented Jun 17, 2025

Uh oh!

Uh oh!

junegunn commented Apr 30, 2025 •

edited

Loading