Skip to content

Federate to an Iceberg JDBC catalog (new JDBC connection type) #5235

Description

@ultramagnetar

Is your feature request related to a problem? Please describe.

Requirement
Polaris supports federating to Hadoop, Hive, and BigQuery metastore catalogs, but not to an
Iceberg JDBC catalog. Many organizations keep Iceberg metadata in a relational database (commonly
PostgreSQL) via Iceberg's JdbcCatalog. We would like Polaris to federate to such a catalog so the
database remains the source of truth for table metadata while Polaris brokers access, policies, and
multi-engine connectivity — consistent with the existing federation types.

Why

  • JDBC/relational Iceberg catalogs are a common deployment; there is currently no way to bring them
    under Polaris governance without migration.
  • It rounds out the federation extension family (hadoop/hive/bigquery → +jdbc).

Out of scope (separate proposal)

  • Sub-catalog (namespace/table-level) RBAC over federated JDBC catalogs.

Environment: Apache Polaris main; JDK 21.

Describe the solution you'd like

Proposed behavior (for discussion)

  • A new JDBC connection type and an optional polaris-extensions-federation-jdbc extension
    (enabled with -PNonRESTCatalogs=JDBC), mirroring the existing federation extensions.
  • A PASSWORD connection authentication type (username + password), with the password offloaded to
    the Polaris user secrets manager (never stored inline), plus support for IMPLICIT auth.
  • Driver-agnostic: no bundled JDBC driver; drivers come from the deployment classpath.

Describe alternatives you've considered

  • Migrate JDBC-catalog metadata into Polaris-managed Iceberg catalogs — rejected: requires a one-time migration and gives up the existing DB as source of truth, which defeats the point of federation.
  • Front the JDBC catalog with an Iceberg REST catalog and use existing REST federation — viable for some, but adds an extra service to deploy/operate and doesn't help orgs that only run JdbcCatalog.
  • Bundle common JDBC drivers in the extension — rejected in favor of sourcing drivers from the deployment classpath, to avoid license/version-pinning issues.

Additional context

I have a working implementation of this along the lines described and am happy to open a PR / discuss the design if there's interest. Related existing federation connection types: Hadoop, Hive, BigQuery.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions