HBASE-26553 OAuth Bearer authentication mech plugin for SASL

[anmolnar]

Adds a new SASL mech plugin for OAuthBearer (JWT) authentication.

• In order to keep the size of this initial patch manageable, the supported workflow is limited: client reads a single JWT token with expiry information from environment variable and authenticates with the server.
• It works similarly to Hadoop delegation tokens, JWT token takes precedence, but if it's missing, the auth provider will fall back to Kerberos.
• Kerberos must be enabled on the cluster, otherwise HBase security is not enabled.

Minimum configuration to enable JWT auth:

Server side:

  <property>
    <name>hbase.server.sasl.provider.extras</name>
    <value>org.apache.hadoop.hbase.security.provider.OAuthBearerSaslServerAuthenticationProvider</value>
  </property>
  <property>
    <name>hbase.security.oauth.jwt.jwks.url</name>
    <value>JWKS download url</value>
  </property>

Client side:

  <property>
    <name>hbase.client.sasl.provider.extras</name>
    <value>org.apache.hadoop.hbase.security.provider.OAuthBearerSaslClientAuthenticationProvider</value>
  </property>
  <property>
    <name>hbase.client.sasl.provider.class</name>
    <value>org.apache.hadoop.hbase.security.provider.OAuthBearerSaslProviderSelector</value>
  </property>

The client also has to be in possession of a valid JWT token which must set via environment variable:

export HBASE_JWT="<base64 encoded token>,<expiry>"

cc @petersomogyi @meszibalu @joshelser @bbeaudreault @Apache9

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 2s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	6m 12s	master passed
+1 💚	checkstyle	1m 1s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	10m 25s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 13s	the patch passed
+1 💚	compile	6m 11s	the patch passed
-0 ⚠️	javac	6m 11s	root generated 3 new + 707 unchanged - 0 fixed = 710 total (was 707)
-0 ⚠️	checkstyle	1m 0s	root: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	8m 0s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌	spotless	0m 12s	patch has 71 errors when running spotless:check, run spotless:apply to fix.
+1 💚	spotbugs	10m 51s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 58s	The patch does not generate ASF License warnings.
		58m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4733
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml
uname	Linux 6546613b29b7 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 06728e5
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-general-check/output/diff-compile-javac-root.txt
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	138 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-resource-bundle hbase-server hbase-examples . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 53s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 41s	master passed
+1 💚	compile	1m 43s	master passed
+1 💚	shadedjars	3m 58s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 54s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 31s	the patch passed
+1 💚	compile	1m 43s	the patch passed
+1 💚	javac	1m 43s	the patch passed
+1 💚	shadedjars	3m 54s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 55s	the patch passed
		_ Other Tests _
-1 ❌	unit	213m 42s	root in the patch failed.
		239m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4733
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5c8a411d2553 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 06728e5
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/testReport/
Max. process+thread count	2599 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-resource-bundle hbase-server hbase-examples . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 56s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 7s	master passed
+1 💚	compile	2m 2s	master passed
+1 💚	shadedjars	5m 38s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	3m 24s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 59s	the patch passed
+1 💚	compile	2m 12s	the patch passed
+1 💚	javac	2m 12s	the patch passed
+1 💚	shadedjars	5m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	3m 42s	the patch passed
		_ Other Tests _
+1 💚	unit	408m 38s	root in the patch passed.
		442m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4733
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7c2a7e7144fd 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 06728e5
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/testReport/
Max. process+thread count	4855 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-resource-bundle hbase-server hbase-examples . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 27s	master passed
+1 💚	compile	9m 43s	master passed
+1 💚	checkstyle	1m 22s	master passed
+1 💚	spotless	0m 57s	branch has no errors when running spotless:check.
+1 💚	spotbugs	15m 2s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 19s	the patch passed
+1 💚	compile	8m 46s	the patch passed
-0 ⚠️	javac	8m 46s	root generated 3 new + 707 unchanged - 0 fixed = 710 total (was 707)
+1 💚	checkstyle	1m 19s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	10m 55s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 52s	patch has no errors when running spotless:check.
+1 💚	spotbugs	15m 57s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 52s	The patch does not generate ASF License warnings.
		82m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4733
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml
uname	Linux b3cdeb7656d8 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 37651ee
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/artifact/yetus-general-check/output/diff-compile-javac-root.txt
Max. process+thread count	138 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-resource-bundle hbase-server hbase-examples . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 40s	master passed
+1 💚	compile	1m 47s	master passed
+1 💚	shadedjars	3m 53s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 53s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	1m 43s	the patch passed
+1 💚	javac	1m 43s	the patch passed
+1 💚	shadedjars	3m 53s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 54s	the patch passed
		_ Other Tests _
+1 💚	unit	249m 44s	root in the patch passed.
		276m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4733
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b765cb021a83 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 37651ee
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/testReport/
Max. process+thread count	4757 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-resource-bundle hbase-server hbase-examples . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 19s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 21s	master passed
+1 💚	compile	1m 36s	master passed
+1 💚	shadedjars	3m 46s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 38s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 14s	the patch passed
+1 💚	compile	1m 37s	the patch passed
+1 💚	javac	1m 37s	the patch passed
+1 💚	shadedjars	3m 45s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 39s	the patch passed
		_ Other Tests _
+1 💚	unit	376m 13s	root in the patch passed.
		400m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4733
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 398b44e09ce9 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 37651ee
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/testReport/
Max. process+thread count	4614 (vs. ulimit of 30000)
modules	C: hbase-common hbase-client hbase-resource-bundle hbase-server hbase-examples . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4733/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[taklwu]

overall if we can reused the reviews in HBASE-26655, HBASE-26665, HBASE-26667 that merged into the feature branch of HBASE-26553, this is just the same commits with minor fixed for the loadJwkSet(). But if you're looking a new round of reviews, please let me know.

below are two minor comments that isn't a blocker for this PR.

1. [nit] HBASE-26655, HBASE-26665, HBASE-26667 was reviewed as part of the feature HBASE-26553 , but why HBASE-26655 does not have a link to HBASE-26553 OAuth Bearer authentication mech plugin for SASL (initial commit) #4019 ?
2. [nit] for the minimum configuration mentioned in the description, don't we also need to configure the environment parameter of HBASE_JWT for the client? maybe we should tell the minimum configuration for the client and server separately in the doc ?

[anmolnar]

Thanks @taklwu , I've updated the minimum configuration in the description with more details.

Sorry for the confusion, I've decided to abandon the feature branch to speed up the rebasing and the process. Please let me know if you think it would be better to go back and continue working on the feature branch instead.

@joshelser

but why HBASE-26655 does not have a link to #4019 ?

I see the link in the jira.