Refactor storage access configuration handling #1504

dimas-b · 2025-05-01T00:39:15Z

This is a step towards supporting non-AWS S3 storage, but this refactoring is relevant to all storage backends.

There is no change to existing behaviours.

Rename PolarisCredentialProperty to IcebergStorageAccessProperty and introduce non-credential properties (as an example for now)
IcebergStorageAccessProperty values are ultimately meant to be produced by PolarisStorageIntegration implementations
Some previous entries in IcebergStorageAccessProperty are not really credential properties, but their treatment is not changed in this PR to maintain exactly the same bahaviour as before.
Add AccessConfig to represent both credential and non-credential properties related to storage access.

collado-mike

Seems like a breaking change without a clear reason. What's the purpose in separating the credentials from extraProperties?

collado-mike · 2025-05-01T06:03:08Z

polaris-core/src/main/java/org/apache/polaris/core/storage/AccessConfig.java

+  Map<String, String> credentials();
+
+  Map<String, String> extraProperties();


What's the need for breaking this out into separate fields? They all end up in the same map for every single caller

The main idea is to isolate credential and non-credential properties in REST API responses. Specifically in IcebergCatalogHandler.

This PR does not cause any functional change, but prepares for exposing properties like s3.endpoint to clients later.

These properties are not always mixed together. PR #1225 brings some distinction in load table responses.

dimas-b · 2025-05-01T17:13:44Z

Seems like a breaking change without a clear reason

@collado-mike : Could you clarify what is broken by this PR? I believe there's no behaviour change.

dimas-b · 2025-05-01T17:16:47Z

What's the purpose in separating the credentials from extraProperties?

The purpose is to allow not mixing credential and non-credential properties in loadTableResponse per #1225

Other renames in this PR are meant to clarity code-level distinction between credentials and non-credential properties related to storage access.

collado-mike · 2025-05-01T18:25:29Z

@collado-mike : Could you clarify what is broken by this PR? I believe there's no behaviour change.

Sorry, I misread the cache code without closely reading the StorageCredentialCacheEntry changes. You're right, there are no behavior changes here.

I understand the reasoning for the change now and I agree.

dimas-b · 2025-05-01T23:38:24Z

@eric-maynard @dennishuo : Please have a look too :)

adnanhemani

I quite like this refactor, it makes lots of sense to do this. One small, stylistic comment.

One minor question, why the name IcebergStorageAccessProperty? I understand that all tables today are Iceberg, but in the future if they were not, would this class still only be relevant to them only?

adnanhemani · 2025-05-02T00:07:45Z

service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

-import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
-import org.apache.polaris.core.storage.PolarisStorageIntegration;
-import org.apache.polaris.core.storage.StorageLocation;
+import org.apache.polaris.core.storage.*;


We should revert this to ensure no wildcard imports.

Good catch - fixed

polaris-core/src/main/java/org/apache/polaris/core/storage/IcebergStorageAccessProperty.java

eric-maynard · 2025-05-02T08:16:41Z

polaris-core/src/main/java/org/apache/polaris/core/storage/IcebergStorageAccessProperty.java

+  AWS_ENDPOINT(String.class, "s3.endpoint", "the S3 endpoint to use for requests", false),
+  AWS_PATH_STYLE_ACCESS(
+      Boolean.class, "s3.path-style-access", "whether to use S3 path style access", false),


eric-maynard · 2025-05-02T08:17:47Z

The semantic changes look good to me, but there's also a rename happening here that looks a bit unrelated to those changes. I'd like to understand the intent behind the rename a little more if possible.

This is a step towards supporting non-AWS S3 storage, but this refactoring is relevant to all storage backends. There is no change to existing behaviours. * Rename PolarisCredentialProperty to IcebergStorageAccessProperty and introduce non-credential properties (as an example for now) * IcebergStorageAccessProperty values are ultimately meant to be produced by PolarisStorageIntegration implementations * Some previous entries in IcebergStorageAccessProperty are not really credential properties, but their treatment is not changed in this PR to maintain exactly the same bahaviour as before. * Add AccessConfig to represent both credential and non-credential properties related to storage access.

adutra · 2025-05-02T20:54:01Z

polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java

+ * <p>Most of these properties are meant to configure Iceberg FileIO objects for accessing data in
+ * storage.
+ */
+public enum StorageAccessProperty {
  AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
  AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
  AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
  AWS_SESSION_TOKEN_EXPIRES_AT_MS(


Is this considered "credential"?

The Key ID, Secret Key, Session Token, and expiry time is definitely together considered a "credential" for AWS. Not sure if there's a reason why you're asking this?

I'd say it is part of the credential bunch of properties.

AWS_SESSION_TOKEN_EXPIRES_AT_MS by itself is definitely not a credential. But it is a property related to credential vending and access to object storage, and so I think having it here makes sense.

adnanhemani

LGTM, thank you!

eric-maynard

LGTM! Excited to see S3-compatible storage support coming together. Thanks for working on this.

ryanovas · 2025-05-05T18:43:29Z

With the other issue arrive S3-compatible storage closed, and this step towards it merged, where can we keep up with the overall progress of S3-compatible storage now?

dimas-b · 2025-05-06T01:00:51Z

Thanks for the reminder @ryanovas . I've just filed a fresh issue for this : #1530 (since #32 has large scope)

@eric-maynard

* Policy Store: PolicyMappingRecord with Persistence Impl (apache#1104) * Spark: Setup repository code structure and build (apache#1190) * Added freshness aware table loading using metadata file location for ETag (apache#1037) * Pulled in iceberg 1.8.0 spec changes for freshness aware table loading and added feature to Polaris * Changed etag support to use entityId:version tuple * fixed getresponse call * Changed etagged response to record and gave default implementation to ETaggableEntity * Made iceberg rest spec docs clearer * Added HTTP Compliant ETag and IfNoneMatch representations and separated persistence from etag logic * Changed ETag to be a record and improved semantics of IfNoneMatch * Fixed semantics of if none match * Removed ETag representation, consolidated in IfNoneMatch * fixed if none match parsing * Added table entity retrieval method to table operations * removed accidental commit of pycache folders * Fixed formatting * Changed to use metadata location hash * Ran formatting * use sha256 * Moved out ETag functions to utility class and removed ETaggedLoadTableResponse * Addressed comments * Fixed IcebergTableLikeEntity package rename * main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.31.0 (apache#1288) * Update LICENSE and NOTICE in the distributions (admin and server) (apache#1258) * Gradle/Quarkus: make imageBuild task depend on jandex (apache#1290) * Core: Clarify the atomicity of BasePersistence methods (apache#1274) * Implement GenericTableCatalogAdapter (apache#1264) * rebase * more fixes * autolint * working on tests * stable test * autolint * polish * changes per review * some changes per review * grants * autolint * changes per review * changes per review * typofix * Improve code-containment and efficiency of etag-aware loading (apache#1296) * Improve code-containment and efficiency of etag-aware loading -Make the hash generation resilient against null metadataLocation -Use getResolvedPath instead of getPassthroughResolvedPath to avoid redundant persistence round-trip -Only try to calculate the etag for comparison against ifNoneMatch if the ifNoneMatch is actually provided * Add strict null-checking at callsites to generateETag, disallow passing null to generator * Add TODO to refactor shared logic for etag generation * Core: Add Endpoints and resource paths for Generic Table (apache#1286) * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.1 (apache#1299) * [JDBC] Part1 : ADD SQL script for Polaris setup (apache#1276) * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1743605859 (apache#1300) * done (apache#1297) * Add Polaris Community Meeting for April 3, 2025 (apache#1304) * Use config-file to define errorprone rule (apache#1233) Also enabled a couple more simple rules, and adding suppressions/fixes for/to the code. The two rules `EqualsGetClass` and `UnusedMethod`, which I think are useful, are not enabled yet, because that would mean actual code changes, which I do not want to do in this PR. The rule `PatternMatchingInstanceof`, introduced in apache#393, is disabled in this PR. It does not work before errorrpone 2.37.0 (via apache#1213) - requires additional changes to enable the rule (see apache#1215). * Add Yun as a contributor (apache#1310) * Refactor CatalogHandler to comply with ErrorProne rules (apache#1312) Fix the CI error after apache#1233 * Implement PolicyCatalog Stage 1: CRUD + ListPolicies (apache#1294) * main: Update dependency io.opentelemetry:opentelemetry-bom to v1.49.0 (apache#1316) * main: Update docker.io/jaegertracing/all-in-one Docker tag to v1.68.0 (apache#1317) * main: Update dependency boto3 to v1.37.28 (apache#1328) * main: Update dependency software.amazon.awssdk:bom to v2.31.16 (apache#1329) * Make `BasePolaritsMetaStoreManagerTest` and `(Base)ResolverTest` reusable (apache#1308) Moves the test cases into the `Base*` classes and make sure the classes can be reused by other persistence implementations. * main: Update dependency io.opentelemetry.semconv:opentelemetry-semconv to v1.32.0 (apache#1293) * main: Update mockito monorepo to v5.17.0 (apache#1311) * PySpark Update AWS Region (apache#1302) Co-authored-by: Travis Bowen <travis.bowen@snowflake.com> * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.2 (apache#1334) * main: Update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.0.3 (apache#1335) * Maven publication: Produce correct `<scm><tag>` in `pom.xml` (apache#1330) `project.scm.tag` in a Maven pom is intended to refer to the SCM (Git) tag. We currently publish `main`, which is incorrect. This change omits the SCM tag for snapshot builds, but emits the Git tag for releases. * Remove `@StaticInitSafe` annotation (apache#1331) There was an issue around mapped configurations having the `@StaticInitSafe` annotation that led to _two_ instances (a "static" one and a "somewhet application-scoped" one) - this was fixed in Quarkus 3.21. One bug in smallrye-config is fixed for Quarkus > 3.21.0, another issue however remains. Since `@StaticInitSafe` annotated configs seem to cause some weird issues, it seems legit to remote that annotation altogether. This approach was [taken in Nessie](projectnessie/nessie#10606) as well. Investigations (via practical experiments) have proven that there's no measurable impact (runtime + heap) when doing this - and that's also been confirmed by Quarkus + Smallrye-config maintainers. Hence this change remotes that annotation from the code base. * Build/Release: Add a "generate digest" task and use for source tarball and Quarkus distributables (apache#1271) * Ensure that digest and signature are generated for both Polaris-Server and admin tar/zip distribution * Move "generate digest" functionality to a Gradle task * main: Update dependency com.google.errorprone:error_prone_core to v2.37.0 (apache#1213) * main: Update Quarkus Platform and Group to v3.21.1 (apache#1291) * main: Update dependency io.netty:netty-codec-http2 to v4.2.0.Final (apache#1301) * Remove unnecessary `clean` and `--no-build-cache` from Gradle invocations (apache#1338) `quarkusAppPartsBuild --rerun` is the right way to force a Docker image build. * Generalize bootstrapping in servers (apache#1313) * Remove `instanceof` checks from `QuarkusProducers`. * Remove the now unused `onStartup` method from `InMemoryPolarisMetaStoreManagerFactory`. * Instead, call the good old `bootstrapRealms` method from `QuarkusProducers`. * Add new config property to control which MetaStore types are bootstrapped automatically (defaults to `in-memory` as before). * There is no bootstrap behaviour change in this PR, only refactorings to simplify code. * Add info log message to indicate when a realm is bootstrapped in runtime using preset credentials. Future enhancements may include pulling preset credentials from a secret manager like Vault for bootstrapping (s discussed in comments on apache#1228). * main: Update actions/stale digest to 816d9db (apache#1341) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4 (apache#1342) * main: Update dependency org.eclipse.persistence:eclipselink to v4.0.6 (apache#1343) * main: Update dependency io.quarkus to v3.21.2 (apache#1344) * main: Update dependency com.google.guava:guava to v33.4.7-jre (apache#1340) Co-authored-by: Robert Stupp <snazy@snazy.de> * Spark: Add Namespaces and View support for SparkCatalog (apache#1332) * Demote technical log messages to DEBUG in PolarisCallContextCatalogFactory (apache#1346) These messages appear to be logging low-level technical details about what is going on in the factory and are not likely to be of interest to most users on a daily basis. * Core/Service: Implement PolicyCatalog Stage 2: detach/attach/getApplicablePolicies (apache#1314) * Spec: Add 'inherited' and 'namespace' Fields to GetApplicablePolicies API Response (apache#1277) * Properly track bootstrappedRealms in InMemoryPolarisMetaStoreManagerFactory (apache#1352) Fixes apache#1351 * Implement GenericTableCatalogAdapter; admin-related fixes (apache#1298) * initial commit: * debugging * some polish * autolint * spec change * bugfix * bugfix * various fixes * another missing admin location * autolint * false by default * fixes per review * autolint * more fixes * DRY * revert small change for a better error * integration test * extra test * autolint * stable * wip * rework subtypes a bit * stable again * autolint * apply new lint rule * errorprone again * adjustments per review * update golden files * add another test * clean up logic in PolarisAdminService * autolint * more fixes per review * format * Update versions in distribution LICENSE and NOTICE (apache#1350) * Spark: Add CreateTable and LoadTable implementation for SparkCatalog (apache#1303) * Add a weigher to the EntityCache based on approximate entity size (apache#490) * initial commit * autolint * resolve conflicts * autolint * pull main * Add multiplier * account for name, too * adjust multiplier * add config * autolint * remove old cast * more tests, fixes per review * add precise weight test * autolint * populate credentials field for loadTableResponse (apache#1225) * populate credentials field for loadTableResponse * spotless * spotless * remove unused hashset * fix merge * fix empty credential case * spotlessApply --------- Co-authored-by: David Lu <dalu@hubspot.com> * main: Update dependency io.smallrye.common:smallrye-common-annotation to v2.12.0 (apache#1355) * Build: Avoid adding duplicated projects for Intelij IDE usage (apache#1333) * main: Update dependency org.junit:junit-bom to v5.12.2 (apache#1354) * main: Update dependency org.apache.commons:commons-text to v1.13.1 (apache#1358) * main: Update dependency boto3 to v1.37.33 (apache#1360) * main: Update dependency software.amazon.awssdk:bom to v2.31.21 (apache#1361) * main: Update dependency io.micrometer:micrometer-bom to v1.14.6 (apache#1362) * main: Update dependency com.google.guava:guava to v33.4.8-jre (apache#1366) * Update LICENSE/NOTICE with latest versions (apache#1364) * Use "clean" LICENSE and NOTICE in published jar artifacts (apache#1292) * main: Update dependency io.projectreactor.netty:reactor-netty-http to v1.2.5 (apache#1372) * Add `Varint` type for variable-length integer encoding (apache#1229) * main: Update docker.io/prom/prometheus Docker tag to v3.3.0 (apache#1375) * Set version to 0.10.0-beta in prepaaration for the next release (apache#1370) * Update the link to OpenAPI in the documentation (apache#1379) * Integration test for Spark Client (apache#1349) * add integration test * add change * add comments * rebase main * update class comments * add base integration * clean up comments * main: Update dependency net.ltgt.gradle:gradle-errorprone-plugin to v4.2.0 (apache#1392) * Add generic table documentations (apache#1374) * add generic table documentation (incomplete) * fix table and spacing * remove documentation in client api since there is no implementation yet * remove spacing * minor fix - proof read * review fix, wording * add generic table documentation (incomplete) * fix table and spacing * remove documentation in client api since there is no implementation yet * remove spacing * minor fix - proof read * review fix, wording * proof read - punctuation fix * change table privilege reference * Unblock test `listNamespacesWithEmptyNamespace` (apache#1289) * Unblock test `listNamespacesWithEmptyNamespace` * Use `containsExactly` to simplify the test * Fix empty namespace behavior * Address comments * Block dropping empty namespace * Improve error messages * Revamp the Quick Start page (apache#1367) * First Draft with AWS * try again * try again * try again * try again * try again * try now * should work * AWS First Draft Complete * ensure file changed * Azure First Draft Complete * Azure First Draft, pt. 2 * Azure Completed * GCP First Draft * GCP Verified * File structure fixed * Remove Trino-specific tutorial * Restructured Quick Start * Addresses minor comments from @eric-maynard * Added reference to Deploying Polaris in Production * Fix MD Link Checker --------- Co-authored-by: Adnan Hemani <adnan.hemani@snowflake.com> * Update README with links to new Quickstart experience (apache#1393) * Update the StorageConfiguration to invoke singleton client objects, a… (apache#1386) * Update the StorageConfiguration to invoke singleton client objects, and add a test * Fix formatting * using guava suppliers * Add aws region * Cleanup and mock test * Spark: Add rest table operations (drop, list, purge and rename etc) for Spark Client (apache#1368) * Initial MVP implementation of Catalog Federation to remote Iceberg REST Catalogs (apache#1305) * Initial prototype of catalog federation just passing special properties into internal properties. Make Resolver federation-aware to properly handle "best-effort" resolution of passthrough facade entities. Targets will automatically reflect the longest-path that we happen to have stored locally and resolve grants against that path (including the degenerate case where the longest-path is just the catalog itself). This provides Catalog-level RBAC for passthrough federation. Sketch out persistence-layer flow for how connection secrets might be pushed down into a secrets-management layer. * Defined internal representation classes for connection config * Construct and initialize federated iceberg catalog based on connection config * Apply the same spec renames to the internal ConnectionConfiguration representations. * Manually pick @XJDKC fixes for integration tests and omittign secrets in response objects * Fix internal connection structs with updated naming from spec PR * Push CreateCatalogRequest down to PolarisAdminService::createCatalog just like UpdateCatalogRequest in updateCatalog. This is needed if we're going to make PolarisAdminService handle secrets management without ever putting the secrets into a CatalogEntity. * Add new interface UserSecretsManager along with a default implementation The default UnsafeInMemorySecretsManager just uses an inmemory ConcurrentHashMap to store secrets, but structurally illustrates the full flow of intended implementations. For mutual protection against a compromise of a secret store or the core persistence store, the default implementation demonstrates storing only an encrypted secret in the secret store, and a one-time-pad key in the returned referencePayload; other implementations using standard crypto protocols may choose to instead only utilize the remote secret store as the encryption keystore while storing the ciphertext in the referencePayload (e.g. using a KMS engine with Vault vs using a KV engine). Additionally, it demonstrates the use of an integrity check by storing a basic hashCode in the referencePayload as well. * Wire in UserSecretsManager to createCatalog and federated Iceberg API handlers Update the internal DPOs corresponding to the various ConnectionConfigInfo API objects to no longer contain any possible fields for inline secrets, instead holding the JSON-serializable UserSecretReference corresponding to external/offloaded secrets. CreateCatalog for federated catalogs containing secrets will now first extract UserSecretReferences from the CreateCatalogRequest, and the CatalogEntity will populate the DPOs corresponding to ConnectionConfigInfos in a secondary pass by pulling out the relevant extracted UserSecretReferences. For federated catalog requests, when reconstituting the actual sensitive secret configs, the UserSecretsManager will be used to obtain the secrets by using the stored UserSecretReferences. Remove vestigial internal properties from earlier prototypes. * Since we already use commons-codec DigestUtils.sha256Hex, use that for the hash in UnsafeInMemorySecretsManager just for consistency and to illustrate a typical scenario using a cryptographic hash. * Rename the persistence-objects corresponding to API model objects with a new naming convention that just takes the API model object name and appends "Dpo" as a suffix; * Use UserSecretsManagerFactory to Produce the UserSecretsManager (#1) * Move PolarisAuthenticationParameters to a top-level property according to the latest spec * Create a Factory for UserSecretsManager * Fix a typo in UnsafeInMemorySecretsManagerFactory * Gate all federation logic behind a new FeatureConfiguration - ENABLE_CATALOG_FEDERATION * Also rename some variables and method names to be consistent with prior rename to ConnectionConfigInfoDpo * Change ConnectionType and AuthenticationType to be stored as int codes in persistence objects. Address PR feedback for various nits and javadoc comments. * Add javadoc comment to IcebergCatalogPropertiesProvider * Add some constraints on the expected format of the URN in UserSecretReference and placeholders for next steps where we'd provide a ResolvingUserSecretsManager for example if the runtime ever needs to delegate to two different implementations of UserSecretsManager for different entities. Reduce the `forEntity` argument to just PolarisEntityCore to make it more clear that the implementation is supposed to extract the necessary identifier info from forEntity for backend cleanup and tracking purposes. --------- Co-authored-by: Rulin Xing <rulin.xing+oss@snowflake.com> Co-authored-by: Rulin Xing <xjdkcsq3@gmail.com> * Add Adnan and Neelesh to collaborators list (apache#1396) * Replace authentication filters with Quarkus Security (apache#1373) * Implement PolicyCatalogHandler and Add Policy Privileges Stage 1: CRUD + ListPolicies (apache#1357) * Add PolicyCatalogHandler and tests * Fix style * Address review comments * Address review comments 2 * fix nit * Remove CallContext.getAuthenticatedPrincipal() (apache#1400) * main: Update dependency info.picocli:picocli-codegen to v4.7.7 (apache#1408) * main: Update dependency com.google.errorprone:error_prone_core to v2.38.0 (apache#1404) * Add Polaris Community Meeting 2025-04-17 (apache#1409) * main: Update dependency boto3 to v1.37.37 (apache#1412) * EclipseLink: add PrimaryKey to policy mapping records JPA model (apache#1403) * Re-instate dependencies between Docker Compose services (apache#1407) * Do not rotate bootstrapped root credentials (apache#1414) * Add Getting Started Button to the Apache Polaris Webshite Homepage (apache#1406) * Core: change to return ApplicablePolicies (apache#1415) * Rename the Snapshot Retention policy (apache#1284) * Rename the Snapshot Retention policy * Resolve comments * Resolve comments --------- Co-authored-by: Yufei Gu <yufei.apache.org> * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.1.0 (apache#1419) * rename snapshotRetention to snashotExpiry (apache#1420) * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1744796716 (apache#1394) * main: Update dependency software.amazon.awssdk:bom to v2.31.26 (apache#1413) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.1.1 (apache#1425) * Fix releaseEmailTemplate task (apache#1384) * Update distributions LICENSE and NOTICE with AWS SDK 2.31.26 update (apache#1423) * Support snapshots=refs (apache#1405) * initial commit * autolint * small revert * rebase * autolint * simpler * autolint * tests * autolint * stable * fix leak * ready for review * improved test * autolint * logic flip again * Update service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com> * Update integration-tests/src/main/java/org/apache/polaris/service/it/env/CatalogApi.java Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com> * adjustments for committed suggestions * autolint --------- Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com> * Remove activatedPrincipalRoles property from AuthenticatedPolarisPrincipal (apache#1410) This seems to be a leftover from when ActiveRolesProvider was introduced. The setter was still used, but the getter wasn't, which hints at the fact that this property can be safely removed. As a bonus, AuthenticatedPolarisPrincipal now becomes immutable, which is imho a very good thing. * Implement PolicyCatalogHandler and Add Policy Privileges Stage 2: AttachPolicy + DetachPolicy (apache#1416) * add auth test for attach/detach * apply formatter * refactor authorizePolicyAttachmentOperation * address comment * better naming * Ship eclipselink and PostgreSQL JDBC driver by default in Polaris distribution (apache#1411) * Fix Connection Config DPOs (apache#1422) * Fix connection config dpos * Run spotlessApply * Doc: Fix the issue that html tags are not working in Hugo (apache#1382) * Implement PolicyCatalogHandler Stage 3: GetApplicablePolicies (apache#1421) * [JDBC] Part2: Add Relational JDBC module (apache#1287) * Bump version to 0.11.0-beta-incubating-SNAPSHOT (apache#1429) * Make entity lookups by id honor the specified entity type (apache#1401) * Make entity lookups by id honor the specified entity type All implementations of `TransactionalPersistence.lookupEntityInCurrentTxn()` are currently ignoring the `typeCode` parameter completely and could potentially return an entity of the wrong type. This can become very concerning during authentication, since a principal lookup could return some entity that is not a principal, and that would be considered a successful authentication. * review * Remove "test" Authenticator (apache#1399) * Propagate SQLException as "caused by" (apache#1430) * Remove logging for DbOps (apache#1433) * Spark: Add regtests for Spark client to test built jars (apache#1402) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.51.0 (apache#1436) * main: Update dependency org.testcontainers:testcontainers-bom to v1.21.0 (apache#1437) * main: Update actions/setup-python digest to a26af69 (apache#1440) * Spark-IT: use correct configurations (apache#1444) ... do not let Spark leak into Quarkus * PolarisRestCatalogIntegrationTest: Always purge generic tables (apache#1443) * Add missing Postgresql dependency (apache#1447) * Add Request Timeouts (apache#1431) * add timeout * add iceberg exception mapping * dont use quarkus bom, disable timeout * nits * Fix sparks sql regtests with up to date config (apache#1454) * Refactor BasePolarisTableOperations & BasePolarisViewOperations (apache#1426) * initial copy paste * Reorder * view copy paste * fixes, polish * stable * yank * CODE_COPIED_TO_POLARIS comments * autolint * update license * typofix * update comments * autolint * Use .sha512 extension instead of -sha512 (apache#1449) * main: Update dependency org.eclipse.microprofile.fault-tolerance:microprofile-fault-tolerance-api to v4.1.2 (apache#1451) * Doc: Update Local Root Principal Credentials in Quickstart (apache#1452) * Update the Getting Started Workflow with each Cloud Provider's Blob Storage (apache#1435) * AWS First Draft * Debug * revert typo * Add JQ to docker runtime * Debug, pt2 * debug * debug * Allow Instance Profile Roles * change random suffix * change instance profile to regular IAM roles * AWS Final Draft * Azure First Draft * debug * Azure First Draft * debug * typo * GCP First Try * GCP Complete * GCP Final * add all jars to Spark * refactor * Implement PolicyCatalogAdapter (apache#1438) * Generic Table/Policy Store: Move feature config check to Adapter and some small refactoring (apache#1465) * update refs (apache#1464) * [JDBC] Part3: Plumb JDBC module to Quarkus (apache#1371) * Allow BasePolarisTableOperations to skip refreshing metadata after a commit (apache#1456) * initial commit * fix another test * changes per comments * visibility * changes per review * autolint * oops * main: Update dependency com.fasterxml.jackson:jackson-bom to v2.19.0 (apache#1455) * Doc: Added set custom credentials instruction in README (apache#1461) * Doc: Add policy documentation (apache#1460) * main: Update dependency software.amazon.awssdk:bom to v2.31.30 (apache#1475) * main: Update dependency gradle to v8.14 (apache#1459) * main: Update dependency gradle to v8.14 * fix PR --------- Co-authored-by: Robert Stupp <snazy@snazy.de> * Remove unused class TokenInfoExchangeResponse (apache#1479) This is an oversight from apache#1399. * Upgrade Polaris to Iceberg 1.9.0 (apache#1309) * Doc: Update on access-control policy docs (apache#1472) * main: Update Quarkus Platform and Group (apache#1381) * Added link to the Spark-Jupyter Notebook Getting Started from the main Getting Started Page (apache#1453) * Added link to the Spark-Jupyter Notebook Getting Started from the main Quickstart page * Typo Co-authored-by: Eric Maynard <emaynard@apache.org> * Suggestions as per @eric-maynard's review * Fix Typo --------- Co-authored-by: Eric Maynard <emaynard@apache.org> * [JDBC] Support Policy (apache#1468) * Refactor EntityCache into an interface (apache#1193) * Refactor EntityCache to an interface * fix * spotless * Remove unused PolarisCredentialVendor.validateAccessToLocations() (apache#1480) * Remove unused PolarisCredentialVendor.validateAccessToLocations() * review: remove ValidateAccessResult and comments * Policy Store: Check whether Policy is in use before dropping and support `detach-all` flag (apache#1467) * fix error (apache#1492) * Ensure writeToPolicyMappingRecord update existing record if primary key equals in EclipseLink Persistence Impl (apache#1469) * update PolicyMappingRecord if not exists * update test * add TODO * Eliminate getCurrentContext() call in PolarisAuthorizerImpl (apache#1494) * Add getting-started for Polaris Spark Client with Delta tables (apache#1488) * Fix: Pull Postgres image automatically (apache#1495) * Fix Outdated Information and add Information regarding `docker compose down` to Quickstart (apache#1497) * Fix Outdated Information and Add Information regarding docker compose down to Quickstart * Revision 2 * Remove shutdown from README * typo * Upgrade Iceberg REST Spec to match Iceberg 1.8 (apache#1283) * prep for review * reset * more changes * fixes * github action change * another build change * try api revert * re-all * custom type mappings, rebuild * autolint * polish * yank custom types * update * autolint * wip * Revert build changes * example * autolint * Fix FileIOExceptionsTest to conform to new Iceberg 1.8 API (apache#1501) It looks like after apache#1283, this test no longer compiles as the Iceberg API has changed. I'm not sure how this wasn't caught by CI on that PR itself. * JDBC: Optimize writeEntity calls (apache#1496) * Remove transaction from atomic writes * remove if-else * main: Update registry.access.redhat.com/ubi9/openjdk-21-runtime Docker tag to v1.22-1.1745840590 (apache#1499) * Support for external identity providers (apache#1397) * JDBC: create objects without reflection (apache#1434) * Include quarkus-container-image and README in the binary distributions (apache#1493) * Site: Fix Management and Catalog Spec links (apache#1507) * Lazy iteration over JDBC ResultSet (apache#1487) * refactor * autolint * polish * autolint * changes per review * autolint * unwrapping caller * changes per review * Update distributions LICENSE and NOTICE with artifacts and versions sync (apache#1509) * Avoid using deprecated `NestedField.of()` (apache#1514) * Fix compile warning: unknown enum constant Id.NAME (apache#1513) * Doc: Add getting started with JDBC source (apache#1470) * Site: Add Polaris Spark client webpage under unreleased (apache#1503) * Add new committers (apache#1518) * Docs: Fix the wrong catalog name in `using polaris` page (apache#1471) * fix Signed-off-by: owenowenisme <mses010108@gmail.com> * update docker compose Signed-off-by: owenowenisme <mses010108@gmail.com> --------- Signed-off-by: owenowenisme <mses010108@gmail.com> * main: Update dependency org.apache.commons:commons-configuration2 to v2.12.0 (apache#1481) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.52.1 (apache#1485) * main: Update dependency com.azure:azure-sdk-bom to v1.2.34 (apache#1490) * main: Update docker.io/prom/prometheus Docker tag to v3.3.1 (apache#1510) * Add new committers on website (apache#1521) * main: Update dependency software.amazon.awssdk:bom to v2.31.35 (apache#1524) * fix overlapping menu item on the nav bar (apache#1520) * fix overlapping menu item on the nav bar * prevent dropdowns expanding inside the navbar * Additional refs update for iceberg 1.9.0 (apache#1491) * Additional refs update for iceberg 1.9.0 * Additional refs update for iceberg 1.9.0 * Additional refs update for iceberg 1.9.0 * Fix typo on Pierre's github URL (apache#1527) * Refactor storage access configuration handling (apache#1504) * Refactor storage access configuration handling This is a step towards supporting non-AWS S3 storage, but this refactoring is relevant to all storage backends. There is no change to existing behaviours. * Rename PolarisCredentialProperty to StorageAccessProperty and introduce non-credential properties (as an example for now) * StorageAccessProperty values are ultimately meant to be produced by PolarisStorageIntegration implementations * Some previous entries in StorageAccessProperty are not really credential properties, but their treatment is not changed in this PR to maintain exactly the same bahaviour as before. * Add AccessConfig to represent both credential and non-credential properties related to storage access. * [JDBC] : Deprecate EclipseLink (apache#1515) * Auto-bootstrap: add verbose logging (apache#1376) Log explicit messages around auto-bootstrapping and unnecessary/left-over secrets that are (still) available. * Add nightly build GH action to publish SNAPSHOT on Nexus (apache#1383) * Add nightly build GH action to publish SNAPSHOT on Nexus (apache#1383) * Build: Fix `fetchAsfProjectName` and make the publishing extension more flexible (apache#1442) The added flexibility is intended to be ported to the multiple project in the polaris-tools repository. (Follow up of apache#1384) * Poetry v2 (apache#898) * PEP 621 and Poetry v2 * PEP 621 and Poetry v2 * Update min python to 3.9 * Add back flask8 for apache#1096 * Add Integration tests for Delta tables for Spark Client (apache#1500) * main: Update dependency com.google.cloud:google-cloud-storage-bom to v2.52.2 (apache#1536) * main: Update dependency poetry to v2.1.3 (apache#1534) * main: Update dependency io.netty:netty-codec-http2 to v4.2.1.Final (apache#1533) * main: Update dependency boto3 to v1.38.10 (apache#1525) * Fix test failure (apache#1541) * Fix the URL of the KEYS file in the release vote email template (apache#1538) * Event Listeners (apache#922) Implementation of event listeners discussed [here](https://lists.apache.org/thread/03yz5wolkvy8l7rbcwjnqdq1bl8p065v). I decided to keep this implementation generic and not take a dependency on Jakarta Events nor Vertx busses. It's easy to extend this, either within Polaris or in an external PolarisEventListener, and handle events however one wishes. Some high level notes: - PolarisEventListener is the main interface with all the event methods such as `onBeforeRequestRateLimited` - DefaultPolarisEventListener is an empty implementation which allows users to only partially implement event handlers - `polaris.events.type` is the config that lets you specify your event listener implementation * Update metastores.md (apache#1537) * Update metastores.md * Resolve comment. * Resolve comment. --------- Co-authored-by: Yufei Gu <yufei.apache.org> * Doc: Document the Concept of realm (apache#1478) * main: Update dependency boto3 to v1.38.11 (apache#1542) * Fix compile warning: [unchecked] unchecked cast (apache#1544) Use `Class.cast()` instead of implicit cast. * NoSQL: Adopt to "Make entity lookups by id honor the specified entity type (apache#1401)" * NoSQL: Filter on correct subtype * NoSQL: merge/rebase 2025/04/30 * additional merge-relaged changes --------- Signed-off-by: owenowenisme <mses010108@gmail.com> Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: gh-yzou <167037035+gh-yzou@users.noreply.github.com> Co-authored-by: Mansehaj Singh <msehajs@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com> Co-authored-by: Yufei Gu <yufei@apache.org> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Dennis Huo <7410123+dennishuo@users.noreply.github.com> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: Travis Bowen <122238243+travis-bowen@users.noreply.github.com> Co-authored-by: Travis Bowen <travis.bowen@snowflake.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com> Co-authored-by: Juichang Lu <wolflex888@gmail.com> Co-authored-by: David Lu <dalu@hubspot.com> Co-authored-by: gfakbar20 <gfakbar20@gmail.com> Co-authored-by: Liam Bao <90495036+liamzwbao@users.noreply.github.com> Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu> Co-authored-by: Adnan Hemani <adnan.hemani@snowflake.com> Co-authored-by: Neelesh Salian <nssalian@users.noreply.github.com> Co-authored-by: Rulin Xing <rulin.xing+oss@snowflake.com> Co-authored-by: Rulin Xing <xjdkcsq3@gmail.com> Co-authored-by: fabio-rizzo-01 <fabio.rizzocascio@jpmorgan.com> Co-authored-by: Pierre Laporte <pierre@pingtimeout.fr> Co-authored-by: Richard Liu <35082658+RichardLiu2001@users.noreply.github.com> Co-authored-by: Michael Collado <40346148+collado-mike@users.noreply.github.com> Co-authored-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: Eric Maynard <emaynard@apache.org> Co-authored-by: Andrew Guterman <andrew.guterman1@gmail.com> Co-authored-by: MonkeyCanCode <yongzheng0809@gmail.com>

dimas-b requested review from adutra, ashvina, dennishuo, eric-maynard, jackye1995, jbonofre, vvcephei and collado-mike as code owners May 1, 2025 00:39

github-project-automation bot added this to Basic Kanban Board May 1, 2025

dimas-b requested review from snazy, RussellSpitzer, takidau, MonkeyCanCode, flyrain and ebyhr as code owners May 1, 2025 00:39

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board May 1, 2025

dimas-b mentioned this pull request May 1, 2025

[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

Closed

10 tasks

collado-mike reviewed May 1, 2025

View reviewed changes

collado-mike previously approved these changes May 1, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 1, 2025

adnanhemani suggested changes May 2, 2025

View reviewed changes

github-project-automation bot moved this from Ready to merge to PRs In Progress in Basic Kanban Board May 2, 2025

eric-maynard reviewed May 2, 2025

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/storage/IcebergStorageAccessProperty.java Outdated Show resolved Hide resolved

eric-maynard reviewed May 2, 2025

View reviewed changes

dimas-b added 2 commits May 2, 2025 15:34

review: fix imports

60c2a14

review: rename to StorageAccessProperty

fa3e82a

dimas-b dismissed collado-mike’s stale review via fa3e82a May 2, 2025 19:34

dimas-b force-pushed the refactor-access-config branch from b9755b3 to fa3e82a Compare May 2, 2025 19:34

adutra reviewed May 2, 2025

View reviewed changes

adnanhemani approved these changes May 2, 2025

View reviewed changes

eric-maynard approved these changes May 3, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 3, 2025

dimas-b merged commit 4db7998 into apache:main May 5, 2025
6 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board May 5, 2025

dimas-b deleted the refactor-access-config branch May 5, 2025 17:08

		Map<String, String> credentials();

		Map<String, String> extraProperties();

Refactor storage access configuration handling #1504

Refactor storage access configuration handling #1504

Uh oh!

Conversation

dimas-b commented May 1, 2025

Uh oh!

collado-mike left a comment

Choose a reason for hiding this comment

Uh oh!

collado-mike May 1, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b May 1, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b commented May 1, 2025

Uh oh!

dimas-b commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

collado-mike commented May 1, 2025

Uh oh!

dimas-b commented May 1, 2025

Uh oh!

adnanhemani left a comment

Choose a reason for hiding this comment

Uh oh!

adnanhemani May 2, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b May 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eric-maynard May 2, 2025

Choose a reason for hiding this comment

Uh oh!

eric-maynard commented May 2, 2025

Uh oh!

adutra May 2, 2025

Choose a reason for hiding this comment

Uh oh!

adnanhemani May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b May 2, 2025

Choose a reason for hiding this comment

Uh oh!

eric-maynard May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adnanhemani left a comment

Choose a reason for hiding this comment

Uh oh!

eric-maynard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryanovas commented May 5, 2025

Uh oh!

dimas-b commented May 6, 2025

Uh oh!

Uh oh!

dimas-b commented May 1, 2025 •

edited

Loading

adnanhemani May 2, 2025 •

edited

Loading

eric-maynard May 3, 2025 •

edited

Loading