Support for external identity providers

[adutra]

Fixes #336 and #976.

Overview

This change is globally backwards-compatible, there are no configuration breaking changes and the default server behavior is the same as today (internal authentication only).

The main interfaces are also unchanged – but some implementations changed.

TODOs

Things to do in follow-up PRs:

• Integration tests (these will be tricky to implement, therefore I prefer to tackle that later)
• Helm chart changes
• Documentation

Authentication Types

To support external identity providers, the notion of "authentication type" has been introduced. The following values are supported:

• internal: Polaris internal authentication only (default)
• external: Polaris external authentication only (using Quarkus OIDC)
  • When this type is enabled globally or for a realm, the realm's internal token endpoint is disabled and returns 501.
• mixed: Polaris internal and external authentication (using Quarkus OIDC)
  • internal is tried first, then external

The default authentication type is internal, but can be either changed globally or overridden per realm.

The authentication type is configured in the application.properties file as below:

# Default authentication type
polaris.authentication.type=internal

# Authentication type overrides per realm
polaris.authentication.realm1.type=external
polaris.authentication.realm2.type=mixed

See below for details of each authentication type.

Configuration Changes

The polaris.authentication configuration section remains backwards-compatible, but everything now can be overridden per realm.

The following options are effective only when using internal auth (can be overridden in per realm):

• polaris.authentication.token-service.*
• polaris.authentication.token-broker.*

The polaris.oidc configuration section is new and is used to further configure the OIDC tenants. See below for examples.

Authenticator Changes

The Authenticator interface remains the same, but BasePolarisAuthenticator has been modified and became DefaultAuthenticator.

This is because Quarkus Security distinguishes two phases of authentication:

1. Extract the credentials (i.e., decode the token)
2. Authenticate the credentials

Therefore, the Authenticator is not responsible anymore for decoding the token, but only for authenticating it.

A new PrincipalCredential interface has been introduced to hold the principal ID, name and roles, and pass it to the authenticator. The existing DecodedToken interface, which is used for internal authentication, is now a sub-type of PrincipalCredential.

The default logic for authenticating the principal, i.e. by a metastore lookup by ID or by name, remains the same.

Token Broker Changes

Token brokers now need to be injected directly, instead of injecting their factories. They are now request-scoped beans.

As a bonus, token brokers can now be configured per realm, i.e., one realm could use RSA key pairs, and another one shared secrets.

Authentication Workflows in Detail

Internal

• InternalAuthenticationMechanism handles the authentication header
  • Calls TokenBroker to decode the token and creates a PrincipalCredential
• InternalIdentityProvider creates a first SecurityIdentity with the PrincipalCredential
• AuthenticatingAugmentor authenticates the PrincipalCredential
  • Calls Authenticator.authenticate()
  • Sets AuthenticatedPolarisPrincipal as the SecurityIdentity's principal
• ActiveRolesAugmentor sets the active roles in the SecurityIdentity
  • Calls ActiveRolesProvider

External

• [Quarkus OIDC] OidcAuthenticationMechanism handles the authentication header
• [Quarkus OIDC] OidcIdentityProvider creates a first SecurityIdentity with the token
• OidcTenantResolvingAugmentor resolves the Polaris OIDC tenant configuration to use and store it as an attribute in the SecurityIdentity
• PrincipalAuthInfoAugmentor maps the JWT claims to a PrincipalAuthInfo and adds it to the SecurityIdentity
• AuthenticatingAugmentor authenticates the PrincipalAuthInfo
  • Calls Authenticator.authenticate()
  • Sets AuthenticatedPolarisPrincipal as the SecurityIdentity's principal
• ActiveRolesAugmentor sets the active roles in the SecurityIdentity
  • Calls ActiveRolesProvider

Mixed

• InternalAuthenticationMechanism handles the authentication header
  • Calls TokenBroker to decode the token
• If the TokenBroker validates the token:
  • InternalIdentityProvider creates a first SecurityIdentity with the token as a PolarisCredential
• Otherwise:
  • InternalAuthenticationMechanism yields to the OIDC mechanism
  • [Quarkus OIDC] OidcAuthenticationMechanism handles the authentication header
  • [Quarkus OIDC] OidcIdentityProvider creates a first SecurityIdentity with the token
  • OidcTenantResolvingAugmentor resolves the Polaris OIDC tenant configuration to use and store it as an attribute in the SecurityIdentity
  • PrincipalAuthInfoAugmentor maps the JWT claims to a PrincipalAuthInfo and adds it to the SecurityIdentity
• For both cases: AuthenticatingAugmentor authenticates the PrincipalAuthInfo
  • Calls Authenticator.authenticate()
  • Sets AuthenticatedPolarisPrincipal as the SecurityIdentity's principal
• ActiveRolesAugmentor sets the active roles in the SecurityIdentity
  • Calls ActiveRolesProvider

OIDC Principal Mapping

Main interface: org.apache.polaris.service.quarkus.auth.external.mapping.PrincipalMapper.

Currently, the following polaris.oidc options are available:

• polaris.oidc.principal-mapper.type: the Principal mapper implementation to use
• polaris.oidc.principal-mapper.id-claim-path: the claim path to the principal id in the JWT token, e.g. polaris/principal_id
• polaris.oidc.principal-mapper.name-claim-path: the claim path to the principal name in the JWT token, e.g. polaris/principal_name

They can be overridden per OIDC tenant (this is different from the realm!). The OIDC tenants will be looked up in the quarkus.oidc configuration section, e.g.:

quarkus.oidc.oidc-tenant1.auth-server-url=http://localhost:8080/realms/polaris
quarkus.oidc.oidc-tenant1.client-id=client1
quarkus.oidc.oidc-tenant1.application-type=service

polaris.oidc.oidc-tenant1.principal-mapper.id-claim-path=polaris/principal_id
polaris.oidc.oidc-tenant1.principal-mapper.name-claim-path=polaris/principal_name

If a tenant is not found in the quarkus.oidc or in the polaris.oidc section, the default OIDC tenant will be used.

Important: the default OIDC tenant is disabled by default, it must be enabled if you intend to use external auth.

OIDC Principal Roles Mapping

Main interface: org.apache.polaris.service.quarkus.auth.external.mapping.PrincipalRolesMapper.

Roles will be mapped from the JWT by Quarkus OIDC. This is done by setting the quarkus.oidc.roles.role-claim-path property:

quarkus.oidc.roles.role-claim-path=polaris/roles

See https://quarkus.io/guides/security-oidc-bearer-token-authentication#token-claims-and-security-identity-roles for more information.

The roles mapped by Quarkus will be available in SecurityIdentity.getRoles(). On the Polaris side, the PrincipalRolesMapper interface converts those roles to Polaris roles; the following properties are available:

• polaris.oidc.principal-roles-mapper.type: the Principal Roles mapper implementation to use.
• polaris.oidc.principal-roles-mapper.filter: optional regex to filter out roles.
• polaris.oidc.principal-roles-mapper.mappings[0].regex: optional regex to modify role names.
• polaris.oidc.principal-roles-mapper.mappings[0].replacement: replacement string.

These properties can be overridden per realm.

Mapping Examples

Example: for a JWT token like this (some claims omitted for brevity):

{
  "sub": "a663c299-1118-4b35-ac85-f9a4f38d0699",
  "typ": "Bearer",
  "azp": "client1",
  "scope": "profile email",
  "polaris": {
    "roles": [ "PRINCIPAL_ROLE:ALL" ],
    "principal_name": "root",
    "principal_id": 1
  },
  "preferred_username": "service-account-client1",
  "clientAddress": "172.18.0.1",
  "client_id": "client1"
}

You can configure Polaris like this:

quarkus.oidc.roles.role-claim-path=polaris/roles
polaris.oidc.principal-mapper.id-claim-path=polaris/principal_id
polaris.oidc.principal-mapper.name-claim-path=polaris/principal_name

Another example; if the JWT token is like this:

{
  "sub": "1"
  "typ": "Bearer",
  "azp": "client1",
  "scope": "service_admin catalog_admin profile email",
  "preferred_username": "root",
  "clientAddress": "172.18.0.1",
  "client_id": "client1"
}

You can configure Polaris like this:

quarkus.oidc.roles.role-claim-path=scope
polaris.oidc.principal-mapper.id-claim-path=sub
polaris.oidc.principal-mapper.name-claim-path=preferred_username
polaris.oidc.principal-roles-mapper.filter=^(?!profile$|email$).*
polaris.oidc.principal-roles-mapper.mappings[0].regex=^.*$
polaris.oidc.principal-roles-mapper.mappings[0].replacement=PRINCIPAL_ROLE:$0

The resulting Polaris principal roles will be: PRINCIPAL_ROLE:service_admin and PRINCIPAL_ROLE:catalog_admin.

[adutra]

For those willing to test out integration with Keycloak:

1. Start a Keycloak server on port 8080 e.g.: http://localhost:8080/realms/master

2. With keycloak admin UI, configure a client to return the following claims (e.g. using hard-coded claim mappers):
   a. polaris/principal_id
   b. polaris/principal_name
   c. polaris/roles

3. Run Polaris with:

./gradlew :polaris-quarkus-service:quarkusDev \
  -Dpolaris.realm-context.realms=realm1,realm2,realm3 \
  -Dpolaris.bootstrap.credentials="realm1,root,secret;realm2,root,secret;realm3,root,secret" \
  -Dpolaris.authentication.realm2.type=external \
  -Dpolaris.authentication.realm3.type=mixed \
  -Dquarkus.oidc.tenant-enabled=true \
  -Dquarkus.oidc.auth-server-url=http://localhost:8080/realms/master \
  -Dquarkus.oidc.roles.role-claim-path=polaris/roles \
  -Dpolaris.oidc.principal-mapper.id-claim-path=polaris/principal_id \
  -Dpolaris.oidc.principal-mapper.name-claim-path=polaris/principal_name

4. 3 realms will be bootstrapped:
   a. realm1 internal auth
   b. realm2 external auth
   c; realm3 mixed auth

5. Obtain a token from Keycloak:

token=$(curl -v http://localhost:8080/realms/master/protocol/openid-connect/token \
  -d client_id=client1 \
  -d client_secret=s3cr3t \
  -d grant_type=client_credentials | jq -r .access_token)

7. Obtain a token from Polaris (realm1 or realm3 – realm2 will return 501):

token=$(curl -v http://localhost:8181/api/catalog/v1/oauth/tokens \
  -H "Polaris-Realm: realm1" \
  -d client_id=root \
  -d client_secret=secret \
  -d grant_type=client_credentials \
  -d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)

8. Using the token:

curl -v http://localhost:8181/api/catalog/v1/config\?warehouse\=default \
  -H "Polaris-Realm: realm1"  \
  -H "Authorization: Bearer $token"

[adutra]

I'm trying to document things as I go. This is the best knowledge I can get for this principal attribute.

[adutra]

We could enable this conditionally with a ConfigSourceInterceptor that would check if any realm is using external or mixed auth, in which case it makes sense to enable the default OIDC tenant, especially if no other tenant is configured. But I'm leaving this as a follow-up improvement.

[adutra]

FYI this was just moved to org.apache.polaris.service.quarkus.auth.internal.InternalIdentityProvider but git lost track of it.

[adutra]

Unused.

[adutra]

This test would fail without this change because each realm would have a different pair of RSA keys. So switching to shared secret.

[adutra]

Again trying to document things a posteriori.

I don't really see the value of returning an Optional here, as the semantics of returning "empty" conflict with that of throwing an error. But I didn't want to change the method signature without knowing what others think.

[dimas-b]

I tend to prefer exceptions here (for follow-up)

[adutra]

I think we need this method to distinguish the case when the principal is requesting all roles from the case where no roles were found in the token – since both cases translate into an empty set of role names.

[adutra]

But a better solution imho would be to remove roles from this class completely, and let SecurityIdentity.getRoles() expose the roles.

I checked the call sites for getActivatedPrincipalRoles() and it's in fact never used. So it seems that setActivatedPrincipalRoles() could be removed as well.

[adutra]

I removed the method because it doesn't work well with the Resolver. TLDR we can't distinguish "no roles" from "all roles" for now.

[adutra]

I believe this is dangerous and should be changed. For example, if the roles were initially:
["PRINCIPAL_ROLE:catalog-admin', "service-admin"], this would be transformed into: ["catalog-admin', "service-admin"], potentially requesting the service-admin role, but that likely wasn't intentional.

I think it would be safer to discard all roles that don't have the PRINCIPAL_ROLE: prefix.

[collado-mike]

Agree, this should be stream().filter(s -> s.startsWith(PRINCIPAL_ROLE_PREFIX)).map(...)

[dimas-b]

+1 to filter

[adutra]

Okay, let's change this.

[adutra]

BTW I also think that the responsibility of stripping off the prefix should be transferred from this class to DefaultPrincipalRolesMapper. But we can do this in a follow-up PR.

[dimas-b]

Nice improvement 👍

[dimas-b]

maybe make ? a declared type parameter to avoid type casts in sub-types?

[dimas-b]

While the statement is true, Polaris still supports the /token endpoint. Should we really flag this as a problem for production deployments? Resolving this requires using Iceberg 1.9 clients plus an Auth Manager implementation, which may be too much of a burden to roll out ATM.

[adutra]

I don't need Iceberg 1.9 and auth manager, client credentials should still work, although without token refreshes. But OK to remove this.

[dimas-b]

I tend to prefer exceptions here (for follow-up)

[dimas-b]

I believe this is a non-issue. My understanding is that tokens need to support restricting to specific roles. Restricting to no roles at all is hardly useful in Polaris since Principals cannot have access rights granted directly to them.

[adutra]

But that's the issue, if no roles are found in the token, Polaris will assume that the token has ALL of the principal roles available at the time of the request. That's imho concerning.

[dimas-b]

I always thought assuming all roles by default was normal in Polaris. I do not see any enforced restrictions for roles in Polaris-owned tokens. So roles in Polaris tokens always looked like a self-imposed restriction to me.

External tokens would have roles / scopes injected by the IdP, I assume.

@collado-mike : WDYT?

[dimas-b]

+1 to filter

[dimas-b]

What makes this class a "Credential"? How about PrincipalAuthInfo?

[adutra]

From what I inferred from Quarkus Security parlance, a "principal" defines an identity, but a "credential" is any concrete form of principal identity proof, and could be: a string, a JWT token, a TLS certificate, etc. So "credential" seemed appropriate here since this is basically a view of a JWT token.

But OK to change to PrincipalAuthInfo 👍

[dimas-b]

Hmmm, but this interface does not have anything that requires it to be backed by JWT and the methods are only informative, they do not provide any "proof" of identity 🤔

[dimas-b]

Would it be more correct to only parse the token here and delegate validation to InternalIdentityProvider?

With that approach, I guess we will not have condition failure logic... All nuance should be handled by Quarkus, WDYT?

We could have a Polaris-specific JWT claim to identify Polaris tokens, I guess.

[adutra]

Well, the issue is that "parse" and "validate" are basically tied together.

Auth0 has two methods: JWT.decode and JWT.require: the former decodes without validating the signature, the latter decodes and validates.

Currently, JWTBroker.verify() uses JWT.require. We could introduce a new method in TokenBroker, e.g. parse() or decode(), and call JWT.decode there. But I don't know if it's worth the hassle: I bet that decoding the token is just slightly faster than decoding and verifying.

BTW that's why I introduced the MIXED authentication type: so that the extra penalty of decoding the token twice is only paid by realms that opt for that authentication type.

[dimas-b]

Performance-wise current approach is probably easiest and fastest. However, InternalIdentityProvider looks like a no-op in the current state of the code, which looks a bit odd to me. Also, the if in exception handling seems to be coming exactly from the uncertainty of the token's origin.

If we parsed the token first, we could delegate to the right validation code and then, if an validation fails it will always be a certain failure.

Regarding parsing penalty, if the token is external, the first attempt will parse and cause an exception on all call, which would be bigger overhead for the lower priority authenticators, I guess.

[adutra]

However, InternalIdentityProvider looks like a no-op in the current state of the code, which looks a bit odd to me

That is very true, but is intentional: we could short-circuit in InternalAuthenticationMechanism and return a SecurityIdentity straight away, rather than invoking identityProviderManager.authenticate(). In that case, an IdentityProvider is not necessary.

BUT: when the identity provider manager is not invoked, no augmentors are invoked either. I discovered this while testing.

So the TLDR is: we need InternalIdentityProvider, even if it's a passthrough impl for now, because there are augmentors that must be invoked.

Regarding parsing penalty, if the token is external, the first attempt will parse and cause an exception on all call, which would be bigger overhead for the lower priority authenticators, I guess.

True. Let me try to introduce the new TokenBroker method and see if that looks better.

[adutra]

After playing with this a bit, I'd be in favor of leaving this for a follow-up PR. The changes to the TokenBroker interface become a bit invasive imho.

I was trying something like this:

interface TokenBroker {
  DecodedToken decode(String token); // done in InternalAuthenticationMechanism
  void verify(DecodedToken token); // done in InternalIdentityProvider
...
}

But:

1. decode would still throw an exception on every request with an external token;
2. We'd need to leak some auth0 types in the DecodedToken interface in order to avoid re-parsing the token. e.g. DecodedJWT getAuth0Token();

[adutra]

Hmm actually my idea won't even work. We need to verify the signature as well in InternalAuthenticationMechanism. That's the only way to be sure that the token is or isn't internal to Polaris.

[dimas-b]

I was thinking that we could parse without validation in InternalAuthenticationMechanism, check a Polaris-specific property and trust it to choose the internal validation logic. Then, InternalIdentityProvider would finish the validation by checking the signature. If the internal property is not present, validation will be done by OIDC.

In any case, I think it makes sense to defer this.

[adutra]

I was able to achieve something that I think will improve the efficiency of token decoding:

adutra#4

Let me know if I should merge those changes into this PR or wait until this one is merged.

[dimas-b]

Let's do that separately.. This PR is already big and reviewed :)

[dimas-b]

@collado-mike : any more comments from your side? :)

[collado-mike]

package private?

[collado-mike]

Why no constructor injection?

[collado-mike]

I think a lot of these classes can be package private. Any reason to expose them outside of this package?

[adutra]

This one specifically is referenced outside the package, but OK for the others 👍

[collado-mike]

constructor inject?

[collado-mike]

I wonder if the ActiveRolesProvider doesn't need more context about the roles the principal should have. E.g., in the external auth case, the token indicates the roles, whereas in the internal case, we have to look up grant records. How does the role provider know whether to accept the tokens contents or to validate the grants?

[adutra]

That's a very good question. You'll notice that for now only one impl of ActiveRolesProvider exists, which will invariably lookup the grants in the database.

When we have federated principals, my suggestion would be to change this Augmentor by either:

1. Creating one different Augmentor for each authentication type (internal/external)
2. Make the Augmentor use a different impl of ActiveRolesProvider depending on the authentication type.

My preference would be for 2, but in any case, my intention was to tackle that once we have federated principals. Wdyt? Does this plan work for you?

[adutra]

We also need a more robust story around roles mapping. For now, most of the work is done by PrincipalRolesMapper, but some further processing is being done in DefaultAuthenticator, for historical reasons. And AuthenticatedPolarisPrincipal wouldn't need to expose roles anymore. But again I'm leaving that for later, as this PR is already big.

[dimas-b]

Deferring this improvement makes sense to me. Also +1 to option 2 above.

[collado-mike]

This plan sounds ok to me. Using the different ActiveRolesProvider for different authn methods sounds like the right way to go.

[collado-mike]

Looks good! thanks for the changes