-
Notifications
You must be signed in to change notification settings - Fork 251
Fix quickstart doc with docker compose #1610
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
c125b2b
6a4be53
f776509
aa8dc9f
1c4a0b8
37b35eb
841ec2e
f338738
8fd5ff8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,10 +24,10 @@ weight: 200 | |
|
||
Polaris can be deployed via a docker image or as a standalone process. Before starting, be sure that you've satisfied the relevant prerequisites detailed in the previous page. | ||
|
||
## Docker Image | ||
|
||
To start using Polaris in Docker, build and launch Polaris, which is packaged with a Postgres instance, Apache Spark, and Trino. | ||
## Common Setup | ||
Before running Polaris, ensure you have completed the following setup steps: | ||
|
||
1. **Build Polaris** | ||
```shell | ||
cd ~/polaris | ||
./gradlew \ | ||
|
@@ -36,7 +36,20 @@ cd ~/polaris | |
:polaris-quarkus-admin:assemble --rerun \ | ||
-Dquarkus.container-image.tag=postgres-latest \ | ||
-Dquarkus.container-image.build=true | ||
docker compose -f getting-started/eclipselink/docker-compose-postgres.yml -f getting-started/eclipselink/docker-compose-bootstrap-db.yml -f getting-started/eclipselink/docker-compose.yml up | ||
``` | ||
- **For standalone**: Omit the `-Dquarkus.container-image.tag` and `-Dquarkus.container-image.build` options if you do not need to build a Docker image. | ||
|
||
## Running Polaris with Docker | ||
|
||
To start using Polaris in Docker and launch Polaris, which is packaged with a Postgres instance, Apache Spark, and Trino. | ||
|
||
```shell | ||
export ASSETS_PATH=$(pwd)/getting-started/assets/ | ||
export CLIENT_ID=root | ||
export CLIENT_SECRET=s3cr3t | ||
docker compose -p polaris -f getting-started/assets/postgres/docker-compose-postgres.yml \ | ||
-f getting-started/eclipselink/docker-compose-bootstrap-db.yml \ | ||
-f getting-started/eclipselink/docker-compose.yml up | ||
``` | ||
|
||
You should see output for some time as Polaris, Spark, and Trino build and start up. Eventually, you won’t see any more logs and see some logs relating to Spark, resembling the following: | ||
|
@@ -48,24 +61,17 @@ spark-sql-1 | 25/04/04 05:39:38 WARN SparkSQLCLIDriver: WARNING: Direct | |
spark-sql-1 | 25/04/04 05:39:39 WARN RESTSessionCatalog: Iceberg REST client is missing the OAuth2 server URI configuration and defaults to http://polaris:8181/api/catalogv1/oauth/tokens. This automatic fallback will be removed in a future Iceberg release.It is recommended to configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be prepared. This warning will disappear if the OAuth2 endpoint is explicitly configured. See https://github.com/apache/iceberg/issues/10537 | ||
``` | ||
|
||
Finally, set the following static credentials for interacting with the Polaris server in the following exercises: | ||
|
||
```shell | ||
export CLIENT_ID=root | ||
export CLIENT_SECRET=s3cr3t | ||
``` | ||
|
||
The Docker image pre-configures a sample catalog called `quickstart_catalog` that uses a local file system. | ||
|
||
## Running Polaris as a Standalone Process | ||
|
||
You can also start Polaris through Gradle (packaged within the Polaris repository): | ||
|
||
1. **Start the Server** | ||
|
||
Run the following command to start Polaris: | ||
|
||
```shell | ||
cd ~/polaris | ||
# Build the server | ||
./gradlew clean :polaris-quarkus-server:assemble :polaris-quarkus-server:quarkusAppPartsBuild --rerun | ||
# Start the server | ||
./gradlew run | ||
``` | ||
|
||
|
@@ -83,11 +89,6 @@ When using a Gradle-launched Polaris instance in this tutorial, we'll launch an | |
For more information on how to configure Polaris for production usage, see the [docs]({{% relref "../configuring-polaris-for-production" %}}). | ||
|
||
When Polaris is run using the `./gradlew run` command, the root principal credentials are `root` and `secret` for the `CLIENT_ID` and `CLIENT_SECRET`, respectively. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not a blocker: We could actually remove this line. |
||
You can also set these credentials as environment variables for use with the Polaris CLI: | ||
```shell | ||
export CLIENT_ID=root | ||
export CLIENT_SECRET=secret | ||
``` | ||
Comment on lines
-86
to
-90
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we still need this to enable polaris cli. |
||
|
||
### Installing Apache Spark and Trino Locally for Testing | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,12 +21,16 @@ Title: Using Polaris | |
type: docs | ||
weight: 400 | ||
--- | ||
|
||
## Setup | ||
|
||
Define your `CLIENT_ID` & `CLIENT_SECRET` and export them for future use. | ||
|
||
```shell | ||
export CLIENT_ID=YOUR_CLIENT_ID | ||
export CLIENT_SECRET=YOUR_CLIENT_SECRET | ||
``` | ||
|
||
## Defining a Catalog | ||
|
||
In Polaris, the [catalog]({{% relref "../entities#catalog" %}}) is the top-level entity that objects like [tables]({{% relref "../entities#table" %}}) and [views]({{% relref "../entities#view" %}}) are organized under. With a Polaris service running, you can create a catalog like so: | ||
|
@@ -167,7 +171,6 @@ bin/spark-sql \ | |
--conf spark.sql.catalog.quickstart_catalog.client.region=us-west-2 | ||
``` | ||
|
||
|
||
Similar to the CLI commands above, this configures Spark to use the Polaris running at `localhost:8181`. If your Polaris server is running elsewhere, but sure to update the configuration appropriately. | ||
|
||
Finally, note that we include the `iceberg-aws-bundle` package here. If your table is using a different filesystem, be sure to include the appropriate dependency. | ||
|
@@ -176,7 +179,9 @@ Finally, note that we include the `iceberg-aws-bundle` package here. If your tab | |
|
||
Refresh the Docker container with the user's credentials: | ||
```shell | ||
docker compose -f getting-started/eclipselink/docker-compose.yml up -d | ||
docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml stop spark-sql | ||
docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml rm -f spark-sql | ||
docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml up -d --no-deps spark-sql | ||
Comment on lines
+182
to
+184
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here, prefer to have a JDBC example. |
||
``` | ||
|
||
Attach to the running spark-sql container: | ||
|
@@ -237,14 +242,15 @@ org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Principal 'quicksta | |
Refresh the Docker container with the user's credentials: | ||
|
||
```shell | ||
docker compose -f getting-started/eclipselink/docker-compose.yml down trino | ||
docker compose -f getting-started/eclipselink/docker-compose.yml up -d | ||
docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml stop trino | ||
docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml rm -f trino | ||
docker compose -p polaris -f getting-started/eclipselink/docker-compose.yml up -d --no-deps trino | ||
Comment on lines
+245
to
+247
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer to provide jdbc examples instead of eclipseLink ones here, we could do that in a follow-up PR though. |
||
``` | ||
|
||
Attach to the running Trino container: | ||
|
||
```shell | ||
docker exec -it eclipselink-trino-1 trino | ||
docker exec -it $(docker ps -q --filter name=trino) trino | ||
``` | ||
|
||
You may not see Trino's prompt immediately, type ENTER to see it. A few commands that you can try: | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we move this line to the top of the file with the exporting of the CLIENT_ID/SECRET, so that it only needs to be run once?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if I followed, this is the beginning of the quickstart page (https://polaris.apache.org/in-dev/unreleased/getting-started/quickstart/#docker-image). This is only needed for the context of docker compose.
The export of CLIENT_ID/CLIENT_SECRET is invalid I think as the current state (without the env) file, this won't even be able to start.
If I understand correctly, we should consider move export of CLIENT_ID/CLIENT_SECRET to this section (as the current docker compose file has no credential, so it will try to set empty string for root credential (as well as username, which is in-valid).
The export is only needed if user doesn't want to use env file (as env file will load the credential in the updated command). Let me know what you think.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, this comment is in conjunction with the suggestion from the overall review's comment. We should do the following:
Does this make sense?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I got what you mean. I had made some changes to this doc for refactor. Please review.