Initial SigV4 Auth Support for Catalog Federation

[XJDKC]

Introduction

This PR adds SigV4 authentication support for Catalog Federation in Polaris, enabling integrations with services like AWS Glue.

It introduces a complete and extensible credential management flow based on the Apache Polaris Creds Management Proposal. The key design goals are to:

• Manage Polaris service identities across tenants
• Inject service identity info into catalog entities during creation
• Retrieve credentials securely at runtime to connect with external services

The implementation supports both self-managed (single-tenant) and SaaS (multi-tenant) deployments, while ensuring a clean separation between user-supplied authentication parameters and Polaris-managed identities/credentials.

How to review this PR?

This PR touches many test files because a new field was added to PolarisCallContext.

Pls focus on the core logic. You can safely ignore most of the test churn, please concentrate on the new components and flow listed below.

For the class name, I'm open to choose other names to make it clearer.

End-to-End flow

The full flow of accessing remote catalog now looks like:

1. User sends CreateCatalogRequest
   • Includes ConnectionConfigInfoModel with SigV4 auth
   • serviceIdentity field is left null
2. EntityMutationEngine applies registered mutators
   • CatalogEntityConnectionConfigMutator injects ServiceIdentityInfoDpo into the entity using ServiceIdentityRegistry
3. Polaris persists the updated ConnectionConfigInfoDpo
   • Now includes both iamArn (user-facing) and a hidden identityReference for credential lookup
4. Accessing Remote Catalog Service
   • PolarisCredentialManager uses ServiceIdentityRegistry to resolve a ResolvedServiceIdentity
   • Assumes the user-provided role using the resolved identity to obtain AWS temp credentials
   • Initialize RESTCatalog client with the temporary credentials

Key Components Introduced

• ServiceIdentityInfoModel: The model we use to surface polaris service identity to users.
• ServiceIdentityInfoDpo: Persisted identity info injected into the catalog entity (includes hidden identityReference), the dpo won't contain the credential.
• ResolvedServiceIdentity: Encapsulates resolved metadata and credentials needed to interact with cloud services.
• ServiceIdentityRegistry: Interface for managing and resolving service identities.
• DefaultServiceIdentityRegistry: Loads service identities from static Quarkus config (per realm, per identity type).
• CatalogEntityConnectionConfigMutator: An EntityMutator that injects the appropriate service identity based on config.
• EntityMutationEngine: Applies configured mutators to polaris entities at some injection points.
• PolarisCredentialManager / DefaultPolarisCredentialManager: Central interface for retrieving credentials to access external services (initially supports connection config, extensible to storage).

Example Configuration:

Here is the example of configuring service identities for each realm statically. But for real production, service vendor should implement their own ServiceIdentityRegistry to assign/resolve service identity dynamically.

polaris.entity-mutation.mutators=catalog-connection-config

polaris.service-identity.registry-identity.type=default
polaris.service-identity.aws-iam.iam-arn=arn:aws:iam::111122223333:user/polaris-default-user
polaris.service-identity.aws-iam.access-key-id=${POLARIS_DEFAULT_ACCESS_KEY_ID}
polaris.service-identity.aws-iam.secret-access-key=${POLARIS_DEFAULT_SECRET_ACCESS_KEY}
polaris.service-identity.my-realm.aws-iam.iam-arn=arn:aws:iam::111122223333:user/polaris-user
polaris.service-identity.my-realm.aws-iam.access-key-id=${POLARIS_ACCESS_KEY_ID}
polaris.service-identity.my-realm.aws-iam.secret-access-key=${POLARIS_SECRET_ACCESS_KEY}

polaris.credential-manager.type=default

Next Steps:

• Cache the connection credentials.
• Unify storage + connection credential flows under a single manager.

Flowchart

A demo video will come soon

[dennishuo]

It's unfortunate that we haven't had a chance yet to make PolarisBaseEntity immutable, but maybe our naming for this set of classes should be something that more strongly implies producing a "new transformed copy of the entity" rather than make it tempting for anyone to try mutating them in-place.

What does everyone think of naming these instead:

• EntityTransformationEngine/EntityTransformer/TransformationPoint
• EntityDecorationEngine/EntityDecorator/DecorationPoint

?

[dennishuo]

Normally we should only increment the entityVersion if the intermediate version is possibly visible to the outside world. It may or may not end up being harmless to modify the version multiple times during construction of the entity, but this would be inconsistent with other steps of the entity-building process.

On the other side, it's true that it might be nice to have something that represents the complete entity contents as a version or hashCode so that we can compare across different transformations.

For now, we should probably avoid incrementing the entityVersion here for consistency with how other things also still construct parts of the entity during creation/updates without incrementing the version for every intermediate state.

[dennishuo]

For now, unfortunately we didn't quite manage to refactor all of the shared logic nryerrn AtomicOperationMetaStoreManager and TransactionalMetaStoreManagerImpl into a base class yet; the best we have is the prepareToPersistNewEntity hook defined in BaseMetaStoreManager.

This means either this logic needs to be added in both the AtomicOperationMetaStoreManager and TransactionalMetaStoreManagerImpl, or needs to be abstracted into BaseMetaStoreManager somehow while making sure both of those impls invoke the shared logic. persistNewEntity isn't specific to catalog-creation so you'd need an enum on entityType there if you just put this logic in there.

[XJDKC]

Got it, thanks for catching this issue and the suggestion! Let me revise my PR!

[dennishuo]

On the one hand, it's useful to be able to have this kind of generalized transformer/mutator hook, but if the hook is also strongly coupled to particular features, we don't want to require separate configuration of the info injection here vs the core feature (e.g. sigv4) somewhere else.

Especially if we end up having feature configurations for the core feature, then it's important that the feature itself auto-configures itself to be functional. In a way then the classes that define the sigv4 functionality should be what injects the mutator/transformer.

[XJDKC]

Hey Dennis, that's a good point, perhaps we can consider making some transformers tied to the feature parameters rather than the config itself if they are part of the core functionality of the feature.