Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge Polaris catalog code (apache#1)
* Initial commit Co-authored-by: Aihua Xu <aihua.xu@snowflake.com> Co-authored-by: Alvin Chen <alvin.chen@snowflake.com> Co-authored-by: Benoit Dageville <benoit.dageville@snowflake.com> Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Evan Gilbert <evan.gilbert@snowflake.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Co-authored-by: Jonas-Taha El Sesiy <github@elsesiy.com> Co-authored-by: Maninder Parmar <maninder.parmar@snowflake.com> Co-authored-by: Michael Collado <collado.mike@gmail.com> Co-authored-by: Sean Lee <sean.lee@snowflake.com> Co-authored-by: Shannon Chen <shannon.chen@snowflake.com> Co-authored-by: Tyler Jones <tyler.jones@snowflake.com> Co-authored-by: Vivo Xu <git@vivoxu.com> * Add brief description to README * Update package naming to separate snowflake code and oss core, extensions, and service impl * Set up new gradle project structure * Add gradle wrapper * Moved docker files to project root and renamed files and classes to Polaris * Update CI scripts to use new layout * Add missing gradlew file * Updates to READMEs and move manual* scripts to snowflake repository root * Fix SparkIntegrationTest after merge * Fix regtest in polaris application Fix json error messages to return clearer validation causes (apache#272) Extended gradle format in root project to apply to oss (apache#273) Improve error message for invalid json and distinguish from invalid values (apache#274) Update repository references to managed-polaris Removed references and made aws resources configurable Fix references to snowflake reg test resources Update README with instructions on running cloud-specific regtests Copy recommended gradle .gitignore contents Update github actions Add @polaris-catalog/polaris team to codeowners * Merge branch 'managed-polaris' into mcollado-polaris-import Co-authored-by: Dennis Huo <huoisme@gmail.com> * Merged changes into polaris-catalog/polaris-dev Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> * Squashed commit of the following: Co-authored-by: Benoit Dageville <benoit.dageville@snowflake.com> Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Eric Maynard <xeyerericx@gmail.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Co-authored-by: Michael Collado <collado.mike@gmail.com> Co-authored-by: Shannon Chen <shannon.chen@snowflake.com> commit bd256f544c069ff15a7a96ab7f2abc650a2e9812 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Tue Jul 23 23:43:38 2024 +0000 Remove s3gov s3china enums and validate roleArn. Removing the enums because the iceberg spec does not have s3gov or s3china prefix for the url, those are snowflake style supported prefix. commit 855dbb702bdc4fc80ca852b8bf563979e08d63d2 Author: Michael Collado <michael.collado@snowflake.com> Date: Tue Jul 23 10:02:35 2024 -0700 Fix credential vending for view creation (apache#19) Correctly sets vended credentials for view creation commit 0429e6356cd71b3908600b6c5c17f82493f1d37d Author: Eric Maynard <eric.maynard@snowflake.com> Date: Tue Jul 23 09:49:20 2024 -0700 This PR implements a basic CLI for Polaris, supporting simple commands like: ``` polaris catalogs list polaris catalogs create --type --storage-type s3 --default-base-location s3://my-bucket --role-arn ${ARN} polaris principals update emaynard --property foo=bar --property e=mc2 polaris privileges --catalog my_cat --catalog-role my_role namespace grant --namespace a.b.c TABLE_READ_DATA polaris privileges --catalog my_cat --catalog-role my_role table revoke --namespace a.b.c --table t1 TABLE_READ_DATA ``` commit 01d4c294e6f8b3e77bf205af00ea2e1dbef0d362 Author: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Date: Mon Jul 22 11:12:29 2024 -0700 Service Bootstrap (Part 2): we are removing bootstrap code in init methods and updates to In-Memory store (apache#8) Changing bootstrap logic, moving bootstrap code to a separate method and only use it during service bootstrapping and first time initialization. So moving forward we will not call bootstrap during SessionManager init code as it used to be, as this will be destructive if service gets restarted. For InMemory Store we have special handling and doing bootstrap on a very first initialization of SessionManager for a given realm. And it makes sense as we can't use our custom dropwizard Bootstrap command for bootstrapping in-memory store (as in-memory store is only valid and available during server process lifetime) commit 2c7f3c43c557e521d7177a4d7dd44157147f0a0c Author: Dennis Huo <dennis.huo@snowflake.com> Date: Fri Jul 19 23:33:05 2024 +0000 Defense-in-depth - make FileIO impl come from StorageConfigurationInfo (apache#15) Description Rather than specifying ResolvingFileIO, we can be more explicit about the FileIO impl we're allowing. Also only allow custom specification of FileIO in test environments using a feature config flag. Even if there are valid FileIO customizations customers could specify, we have only really vetted the enumerated list of impls, so for example we don't want a customer to be able to force Polaris to try to use Hadoop's S3AFileSystem for S3 files even if it "might" work. This in conjunction with omitting `FILE` from SUPPORTED_CATALOG_STORAGE_TYPES for managed environments (https://github.com/snowflakedb/polaris-k8-config/pull/116/files) ensures we won't have a FileIO impl that's capable of reading unexpected files. commit 498861114994b0508efdbdd2167918be5517f4cb Merge: cf07ac0 c100175 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 13:41:02 2024 -0700 Merge branch 'main' into mcollado-update-aws-region commit cf07ac099644b96f93026b209c9938243c1cce18 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 13:38:22 2024 -0700 Stop setting AWS_REGION env and use client.region spark config in tests commit c10017521145e138ae5cdd903d7d51b4bee9e82c Merge: b1de84a d2df00f Author: Eric Maynard <eric.maynard@snowflake.com> Date: Fri Jul 19 12:43:15 2024 -0700 Merge pull request apache#12 from snowflakedb/confirm-warehouse-non-null commit b1de84ad47f6bdf5be4318d4664767dfc33bb5a0 Merge: 504dcc0 1f79e19 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 09:25:07 2024 -0700 Merge branch 'main' into mcollado-view-tests commit d4c58a6a19756078309229c1de4dbf5f737dbdd0 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Thu Jul 18 02:58:52 2024 -0700 cross region support commit 504dcc05bb33e686f5765e5b2d91aa4dcfe2e5d1 Author: Michael Collado <michael.collado@snowflake.com> Date: Fri Jul 19 00:00:57 2024 -0700 fix regtest failures commit b7ed5d27e2d71708977cc6fe7eac3ab10e8d9836 Author: Michael Collado <michael.collado@snowflake.com> Date: Thu Jul 18 21:52:46 2024 -0700 Add reg tests to verify view support * Squashed commit of the following: commit 4fb3b6c19a8a8a4961b777ad32dbe1b87d5efe94 Author: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Date: Thu Jul 25 14:02:30 2024 -0700 Adding annotation and enforcing size limits for Principal, Role, Catalog and Catalog Role names. Also blocking "SYSTEM$" prefix from being used in names. Adding case-insensitive regex rule to block "SYSTEM$" commit 2fcc2898ea038c074fed075cdc7ff62e4884e76a Author: Alvin Chen <alvin.chen@snowflake.com> Date: Thu Jul 25 11:28:00 2024 -0700 Replace Dropwizard Metrics with Micrometer (apache#18) <!-- Please describe your change here and remove this comment --> Since the current Dropwizard Metric library 4.x doesn't support adding custom labels to metrics, we cannot define per-account metrics in order As a result, we're migrating to Micrometer metrics to support custom tagging and align with the metric implementations Major changes by component - `PolarisMetricRegistry` - defines caching for timers and error counters as well as abstracts away the creation of two separate metrics, one with and one without the `account` tag - `TimedApplicationEventListener` - an implementation of the Jersey ApplicationEventListener to listen on requests invoking methods with `@TimedApi` annotation, and handles logic of timing resource/counting errors on success/failure cases respectively - `IcebergMappedException` - removed the original logic for counting errors since the code is now centralized in the above two classes ## Test Manual tested by calling the /metrics endpoint. Following is the result of one successful and one failure invoke of the /oauth endpoint. Note that the timer produces a `summary` and a `gauge`, and doesn't get incremented on failure cases. ``` % curl http://localhost:8182/metrics # HELP polaris_OAuth2Api_getToken_error_total # TYPE polaris_OAuth2Api_getToken_error_total counter polaris_OAuth2Api_getToken_error_total{HTTP_RESPONSE_CODE="401"} 1.0 # HELP polaris_OAuth2Api_getToken_error_realm_total # TYPE polaris_OAuth2Api_getToken_error_realm_total counter polaris_OAuth2Api_getToken_error_realm_total{HTTP_RESPONSE_CODE="401",REALM_ID="testpolaris"} 1.0 # HELP polaris_OAuth2Api_getToken_realm_seconds # TYPE polaris_OAuth2Api_getToken_realm_seconds summary polaris_OAuth2Api_getToken_realm_seconds_count{REALM_ID="testpolaris"} 1 polaris_OAuth2Api_getToken_realm_seconds_sum{REALM_ID="testpolaris"} 0.384 # HELP polaris_OAuth2Api_getToken_realm_seconds_max # TYPE polaris_OAuth2Api_getToken_realm_seconds_max gauge polaris_OAuth2Api_getToken_realm_seconds_max{REALM_ID="testpolaris"} 0.384 # HELP polaris_OAuth2Api_getToken_seconds # TYPE polaris_OAuth2Api_getToken_seconds summary polaris_OAuth2Api_getToken_seconds_count 1 polaris_OAuth2Api_getToken_seconds_sum 0.384 # HELP polaris_OAuth2Api_getToken_seconds_max # TYPE polaris_OAuth2Api_getToken_seconds_max gauge polaris_OAuth2Api_getToken_seconds_max 0.384 # HELP polaris_persistence_loadEntity_realm_seconds # TYPE polaris_persistence_loadEntity_realm_seconds summary polaris_persistence_loadEntity_realm_seconds_count{REALM_ID="testpolaris"} 1 polaris_persistence_loadEntity_realm_seconds_sum{REALM_ID="testpolaris"} 0.041 # HELP polaris_persistence_loadEntity_realm_seconds_max # TYPE polaris_persistence_loadEntity_realm_seconds_max gauge polaris_persistence_loadEntity_realm_seconds_max{REALM_ID="testpolaris"} 0.041 # HELP polaris_persistence_loadEntity_seconds # TYPE polaris_persistence_loadEntity_seconds summary polaris_persistence_loadEntity_seconds_count 1 polaris_persistence_loadEntity_seconds_sum 0.041 # HELP polaris_persistence_loadEntity_seconds_max # TYPE polaris_persistence_loadEntity_seconds_max gauge polaris_persistence_loadEntity_seconds_max 0.041 ``` commit 5abee21b07be00f5f3b18faabe61fb88ecec37e0 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Thu Jul 25 17:14:09 2024 +0000 select view hangs in remote polaris because iceberg SDK could not initialize the s3client since it is missing credentials. It works locally because the SDK S3client initialization work if your local environment have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set, and our dev environment does have these two variables set, so it was not using vending scopedcreds. This PR does below things: 1. add scoped creds to the fileIO when select view 2. stops retry for more cases, the `select view` hangs because it keeps retying commit 85d41bcbab30c9fc3fad56dea83f80e8146ee79c Author: Eric Maynard <eric.maynard@snowflake.com> Date: Wed Jul 24 16:33:57 2024 -0700 In this PR, I've regenerated the Python clients from the spec by following the steps [here](https://github.com/snowflakedb/managed-polaris/tree/main/polaris/regtests#python-tests). I ran: ``` docker run --rm \ -v ${PWD}:/local openapitools/openapi-generator-cli generate \ -i /local/spec/polaris-management-service.yml \ -g python \ -o /local/regtests/client/python --additional-properties=packageName=polaris.management --additional-properties=apiNamePrefix=polaris docker run --rm \ -v ${PWD}:/local openapitools/openapi-generator-cli generate \ -i /local/spec/rest-catalog-open-api.yaml \ -g python \ -o /local/regtests/client/python --additional-properties=packageName=polaris.catalog --additional-properties=apiNameSuffix="" --additional-properties=apiNamePrefix=Iceberg ``` commit 485d99c89abd7b7c3690f45d96a5043a47032ba3 Author: Eric Maynard <eric.maynard@snowflake.com> Date: Wed Jul 24 11:27:21 2024 -0700 This PR introduces quickstart documentation and adds a basic structure for OSS docs. commit 4310980aecf81cc23bbf583cfb6c360ca738a788 Author: Shannon Chen <shannon.chen@snowflake.com> Date: Wed Jul 24 17:38:14 2024 +0000 Stop retry 403 Access Denied error (apache#22) commit 95acd5b3e7983b89d47a915c62ac5bb247730313 Author: Benoit Dageville <59930187+sfc-gh-bdagevil@users.noreply.github.com> Date: Tue Jul 23 22:15:34 2024 -0700 * Fix readme statement and snowflake reference in PolarisDefaultDiagServiceImpl --------- Co-authored-by: Daniel Myers <jdanielmyers@gmail.com> Co-authored-by: Anna Filippova <7892219+annafil@users.noreply.github.com> Co-authored-by: Michael Collado <michael.collado@snowflake.com> Co-authored-by: Aihua Xu <aihua.xu@snowflake.com> Co-authored-by: Alvin Chen <alvin.chen@snowflake.com> Co-authored-by: Benoit Dageville <benoit.dageville@snowflake.com> Co-authored-by: Dennis Huo <huoisme@gmail.com> Co-authored-by: Evan Gilbert <evan.gilbert@snowflake.com> Co-authored-by: Evgeny Zubatov <evgeny.zubatov@snowflake.com> Co-authored-by: Jonas-Taha El Sesiy <github@elsesiy.com> Co-authored-by: Maninder Parmar <maninder.parmar@snowflake.com> Co-authored-by: Sean Lee <sean.lee@snowflake.com> Co-authored-by: Shannon Chen <shannon.chen@snowflake.com> Co-authored-by: Tyler Jones <tyler.jones@snowflake.com> Co-authored-by: Vivo Xu <git@vivoxu.com>
- Loading branch information
16 people
committed
Jul 30, 2024
1 parent
dbcbb79
commit 88fbb9a