HBASE-29666:Additional key management APIs #7460

haridsv · 2025-11-12T08:25:07Z

This PR implements the rest of the planned key management APIs and adds full test coverage for these APIs. It also fixes a few bugs and other minor issues and also makes some generic code quality improvements on top of the open PR #7421.

HBASE-29666

…eak a bad dependency pattern

…oto so that they can be reused for upcoming Admin APIs

…perly

…taCacheEntryOnServers and clearManagedKeyDataCacheOnAllServers to clearManagedKeyDataCacheOnServers

…erage

…or issues flagged in CI build

Apache-HBase · 2025-11-25T08:08:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-29368-key-management-feature Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 24s		HBASE-29368-key-management-feature passed
+1 💚	compile	6m 56s		HBASE-29368-key-management-feature passed
+1 💚	checkstyle	3m 3s		HBASE-29368-key-management-feature passed
+1 💚	spotbugs	6m 40s		HBASE-29368-key-management-feature passed
+1 💚	spotless	0m 49s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 7s		the patch passed
+1 💚	compile	6m 42s		the patch passed
+1 💚	cc	6m 42s		the patch passed
-0 ⚠️	javac	0m 44s	/results-compile-javac-hbase-common.txt	hbase-common generated 1 new + 194 unchanged - 1 fixed = 195 total (was 195)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 5s	/results-checkstyle-hbase-server.txt	hbase-server: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
-0 ⚠️	rubocop	0m 29s	/results-rubocop.txt	The patch generated 15 new + 44 unchanged - 1 fixed = 59 total (was 45)
+1 💚	spotbugs	7m 3s		the patch passed
+1 💚	hadoopcheck	11m 55s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	hbaseprotoc	2m 31s		the patch passed
-1 ❌	spotless	0m 41s		patch has 23 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
+1 💚	asflicense	1m 15s		The patch does not generate ASF License warnings.
		67m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7460
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc rubocop
uname	Linux 06e725ca4894 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29368-key-management-feature / `ac5a3eb`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/7/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/7/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 rubocop=1.37.1
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-25T11:48:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-29368-key-management-feature Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 35s		HBASE-29368-key-management-feature passed
+1 💚	compile	2m 55s		HBASE-29368-key-management-feature passed
+1 💚	javadoc	1m 51s		HBASE-29368-key-management-feature passed
+1 💚	shadedjars	6m 21s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 14s		the patch passed
+1 💚	compile	2m 55s		the patch passed
+1 💚	javac	2m 55s		the patch passed
+1 💚	javadoc	0m 10s		hbase-protocol-shaded in the patch passed.
-0 ⚠️	javadoc	0m 15s	/results-javadoc-javadoc-hbase-common.txt	hbase-common generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
+1 💚	javadoc	0m 17s		hbase-client in the patch passed.
+1 💚	javadoc	0m 27s		hbase-server generated 0 new + 63 unchanged - 2 fixed = 63 total (was 65)
+1 💚	javadoc	0m 31s		hbase-thrift in the patch passed.
+1 💚	javadoc	0m 10s		hbase-shell in the patch passed.
+1 💚	shadedjars	6m 12s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 34s		hbase-protocol-shaded in the patch passed.
+1 💚	unit	2m 56s		hbase-common in the patch passed.
+1 💚	unit	1m 28s		hbase-client in the patch passed.
+1 💚	unit	232m 32s		hbase-server in the patch passed.
+1 💚	unit	6m 39s		hbase-thrift in the patch passed.
+1 💚	unit	6m 52s		hbase-shell in the patch passed.
		287m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7460
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux e040a807b305 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29368-key-management-feature / `ac5a3eb`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/7/testReport/
Max. process+thread count	4190 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/7/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-25T14:56:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	5m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-29368-key-management-feature Compile Tests _
+0 🆗	mvndep	0m 17s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 58s		HBASE-29368-key-management-feature passed
+1 💚	compile	8m 13s		HBASE-29368-key-management-feature passed
+1 💚	checkstyle	3m 25s		HBASE-29368-key-management-feature passed
+1 💚	spotbugs	8m 18s		HBASE-29368-key-management-feature passed
+1 💚	spotless	1m 5s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 42s		the patch passed
+1 💚	compile	8m 16s		the patch passed
+1 💚	cc	8m 16s		the patch passed
-0 ⚠️	javac	0m 54s	/results-compile-javac-hbase-common.txt	hbase-common generated 2 new + 193 unchanged - 2 fixed = 195 total (was 195)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 8s		The patch passed checkstyle in hbase-protocol-shaded
+1 💚	checkstyle	0m 18s		The patch passed checkstyle in hbase-common
+1 💚	checkstyle	0m 22s		The patch passed checkstyle in hbase-client
+1 💚	checkstyle	1m 9s		hbase-server: The patch generated 0 new + 9 unchanged - 1 fixed = 9 total (was 10)
+1 💚	checkstyle	1m 12s		The patch passed checkstyle in hbase-thrift
+1 💚	checkstyle	0m 10s		The patch passed checkstyle in hbase-shell
-0 ⚠️	rubocop	0m 21s	/results-rubocop.txt	The patch generated 15 new + 44 unchanged - 1 fixed = 59 total (was 45)
+1 💚	spotbugs	8m 45s		the patch passed
+1 💚	hadoopcheck	13m 46s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	hbaseprotoc	2m 41s		the patch passed
-1 ❌	spotless	0m 46s		patch has 23 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		84m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7460
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc rubocop
uname	Linux 03e5c0ab95f0 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29368-key-management-feature / `8302aa3`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/8/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	71 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/8/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 rubocop=1.37.1
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-11-25T18:17:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
-0 ⚠️	yetus	0m 4s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-29368-key-management-feature Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 7s		HBASE-29368-key-management-feature passed
+1 💚	compile	2m 55s		HBASE-29368-key-management-feature passed
+1 💚	javadoc	1m 49s		HBASE-29368-key-management-feature passed
+1 💚	shadedjars	6m 13s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 17s		the patch passed
+1 💚	compile	2m 54s		the patch passed
+1 💚	javac	2m 54s		the patch passed
+1 💚	javadoc	0m 10s		hbase-protocol-shaded in the patch passed.
-0 ⚠️	javadoc	0m 15s	/results-javadoc-javadoc-hbase-common.txt	hbase-common generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
+1 💚	javadoc	0m 17s		hbase-client in the patch passed.
+1 💚	javadoc	0m 27s		hbase-server generated 0 new + 63 unchanged - 2 fixed = 63 total (was 65)
+1 💚	javadoc	0m 31s		hbase-thrift in the patch passed.
+1 💚	javadoc	0m 9s		hbase-shell in the patch passed.
+1 💚	shadedjars	6m 14s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 34s		hbase-protocol-shaded in the patch passed.
-1 ❌	unit	3m 17s	/patch-unit-hbase-common.txt	hbase-common in the patch failed.
+1 💚	unit	1m 28s		hbase-client in the patch passed.
-1 ❌	unit	232m 17s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
+1 💚	unit	6m 50s		hbase-thrift in the patch passed.
+1 💚	unit	6m 52s		hbase-shell in the patch passed.
		286m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7460
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 74b36142c300 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29368-key-management-feature / `8302aa3`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/8/testReport/
Max. process+thread count	4468 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-thrift hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7460/8/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

haridsv added 12 commits November 5, 2025 14:58

Move a util method from KeyManagementBase to KeyManagementUtils to br…

785e4a8

…eak a bad dependency pattern

Move key management message classes from ManagedKey.proto to HBase.pr…

c769d75

…oto so that they can be reused for upcoming Admin APIs

Add eject and clear methods for cache

e3a5a24

Cursor implementation for eject and clear, first cut

79d3961

Cursor changes for code review round 1

eec3b8f

Misc. fixes

350fe02

Test fixes

c2a52ae

Rename methods for consistency

676cba0

More test coverage

92dfcf7

Refactor test code

32ea3c8

Remove duplicate method

05c4af9

test coverage for cache eject

4a52653

This comment has been minimized.

Sign in to view

haridsv force-pushed the addl-keymgmt-apis branch from 6db38ca to a3b1155 Compare November 13, 2025 12:54

This comment has been minimized.

Sign in to view

haridsv added 13 commits November 14, 2025 20:43

Expect key metadata instead of hash to be able to clear the cache pro…

0f49f26

…perly

Test coverage for eject

f3e690e

Symbol rename for consistency

96dbc43

Cleanup unused method and remove duplication

b45c4f7

Cursor change to add new APIs, unverified

792e7b0

Fixes to comppilation errors

3b65ed4

Ran spotify:apply

c0c97ca

Rename ejectManagedKeyDataCacheEntryOnAllServers to ejectManagedKeyDa…

0213acc

…taCacheEntryOnServers and clearManagedKeyDataCacheOnAllServers to clearManagedKeyDataCacheOnServers

spotless:apply

991dc3f

WIP changes to drop key metadata usage from client side

101c773

Small refactor

b4b4f9b

Simplify the disable flow

3858852

Misc code review fixes to Cursor changes

9b934cf

haridsv added 8 commits November 14, 2025 20:43

Removed redundant RotateSTKResponse

39a0fda

New test coverage by cursor, needs review

d635dad

Misc. manual review changes such as cleanups, refactors and addl. cov…

cdc60e0

…erage

Bug fix for active key lookup

bc1a621

Shell commands for new API and E2E test coverage

2c91e3f

More test coverage

9ee4683

Follow existing test pattern for the change in encryption attribute

771b927

Missing Test annotation

6ce3065

haridsv force-pushed the addl-keymgmt-apis branch from a3b1155 to 6ce3065 Compare November 14, 2025 15:49

This comment has been minimized.

Sign in to view

Fix javadoc errors

9441ca6

This comment has been minimized.

Sign in to view

haridsv added 2 commits November 20, 2025 11:45

WIP manual changes to address gaps in state handling

c7875f9

Stabiliation of test coverage

cd4e9b7

This comment has been minimized.

Sign in to view

haridsv added 3 commits November 24, 2025 14:43

Primarily updated the behavior of disableKeyManagement

6924f45

Additional test coverage by Cursor

26dc9fd

Ran spotless

88cdcae

This comment has been minimized.

Sign in to view

haridsv added 2 commits November 25, 2025 12:03

Better exception handling and logging in refreshKey and other fixes f…

9051ada

…or issues flagged in CI build

Address misc. CI issues

ac5a3eb

Fixed another CI issue

8302aa3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-29666:Additional key management APIs #7460

HBASE-29666:Additional key management APIs #7460

haridsv commented Nov 12, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Nov 25, 2025

Uh oh!

Apache-HBase commented Nov 25, 2025

Uh oh!

Apache-HBase commented Nov 25, 2025

Uh oh!

Apache-HBase commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HBASE-29666:Additional key management APIs #7460

Are you sure you want to change the base?

HBASE-29666:Additional key management APIs #7460

Conversation

haridsv commented Nov 12, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Nov 25, 2025

Uh oh!

Apache-HBase commented Nov 25, 2025

Uh oh!

Apache-HBase commented Nov 25, 2025

Uh oh!

Apache-HBase commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants