Skip to content

docs(kubernetes-gcp-gke): update guide #2859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 23, 2025
Merged

docs(kubernetes-gcp-gke): update guide #2859

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 35 additions & 35 deletions content/docs/02.installation/05.kubernetes-gcp-gke.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,19 @@ title: Kubernetes on GCP GKE with CloudSQL and Cloud Storage
icon: /docs/icons/gcp-gke.svg
---

Deploy Kestra to GCP GKE with CloudSQL as a database backend and Google Cloud Storage as internal storage backend.
Deploy Kestra to GCP GKE with CloudSQL as the database backend and Google Cloud Storage as the internal storage backend.

## Overview
This guide provides detailed instructions for deploying Kestra to Google Kubernetes Engine (GKE) with CloudSQL as database backend, and Google Cloud Storage(GCS) for internal storage.
This guide provides detailed instructions for deploying Kestra to Google Kubernetes Engine (GKE) with CloudSQL as the database backend and Google Cloud Storage (GCS) for internal storage.

**Prerequisites:**
- Basic command line interface skills
- Familiarity with GCP GKE, PostgreSQL, GCS, and Kubernetes
- Basic command-line interface (CLI) skills.
- Familiarity with GCP GKE, PostgreSQL, GCS, and Kubernetes.

## Launch an GKE Cluster
First, login to GCP using `gcloud init`.

Run the following command to create an GKE cluster named `my-kestra-cluster`:
Run the following command to create a GKE cluster named `my-kestra-cluster`:

```shell
gcloud container clusters create my-kestra-cluster --region=europe-west3
Expand Down Expand Up @@ -85,11 +85,11 @@ This configuration links your Kubernetes service account to the GCP service acco
## Launch CloudSQL

1. Go to the [Cloud SQL console](https://console.cloud.google.com/sql/instances).
2. Click on `Choose PostgreSQL` (Kestra also supports MySQL, but PostgreSQL is recommended).
2. Click on **Choose PostgreSQL** (Kestra also supports MySQL, but PostgreSQL is recommended).
3. Put an appropriate Instance ID and password for the admin user `postgres`.
4. Select the latest PostgreSQL version from the dropdown.
5. Choose `Enterprise Plus` or `Enterprise` edition based on your requirements.
6. Choose an appropriate preset among `Production`, `Development` or `Sandbox` as per your requirement.
5. Choose **Enterprise Plus** or **Enterprise** edition based on your requirements.
6. Choose an appropriate preset among **Production**, **Development** or **Sandbox** as per your requirement.
7. Choose the appropriate region and zonal availability.
8. Click create and wait for completion.

Expand All @@ -99,31 +99,31 @@ This configuration links your Kubernetes service account to the GCP service acco

**Enable VM connection to database**

1. Go to the database overview page and click on `Connections` from the left-side navigation menu.
2. Go to the `Networking` tab and click on `Add a Network`.
3. In the New Network section, add an appropriate name like `Kestra VM` and enter your GKE pods IP address range in the Network.
4. Click on `Done` in the section.
5. Click on `Save` on the page.
1. Go to the database overview page and click on **Connections** from the left-side navigation menu.
2. Go to the **Networking** tab and click on **Add a Network**.
3. In the New Network section, add an appropriate name like **Kestra VM** and enter your GKE pods' IP address range in the network.
4. Click on **Done** in the section.
5. Click on **Save** on the page.

![db_connections](/docs/administrator-guide/deployment/gcp-vm/db_connections.png)

![db_add_a_network](/docs/administrator-guide/deployment/gcp-vm/db_create_connection.png)

**Create database user**

1. Go to the database overview page and click on `Users` from the left-side navigation menu.
2. Click on `Add User Account`.
3. Put an appropriate username and password and click on `Add`.
1. Go to the database overview page and click on **Users** from the left-side navigation menu.
2. Click on **Add User Account**.
3. Put an appropriate username and password and click on **Add**.

![db_users](/docs/administrator-guide/deployment/gcp-vm/db_users.png)

![db_user_creation](/docs/administrator-guide/deployment/gcp-vm/db_user_creation.png)

**Create Kestra database**

1. Go to the database overview page and click on `Databases` from the left side navigation menu.
2. Click on `Create Database`.
3. Put an appropriate database name and click on `Create`.
1. Go to the database overview page and click on **Databases** from the left side navigation menu.
2. Click on **Create Database**.
3. Put an appropriate database name and click on **Create**.

**Update Kestra configuration**

Expand All @@ -144,7 +144,7 @@ configuration:
password: <your-password>
```

Also, disable the PostgreSQL pod by changing `enabled` value in the `postgresql` section from `true` to `false` in the same file.
Also, disable the PostgreSQL pod by changing the `enabled` value in the `postgresql` section from `true` to `false` in the same file.
```yaml
postgresql:
enabled: false
Expand All @@ -158,20 +158,20 @@ helm upgrade my-kestra kestra/kestra -f values.yaml

## Prepare a GCS bucket

By default, Minio pod is being used as storage backend, and internal storage is implemented using the local file system. This section guides you how to change the storage backend to Cloud Storage to ensure more reliable, durable, and scalable storage.
By default, Minio pod is being used as storage backend, and internal storage is implemented using the local file system. This section guides you on how to change the storage backend to Cloud Storage to ensure more reliable, durable, and scalable storage.

1. Go to the Cloud Storage console and create a bucket.
2. Go to IAM and select `Service Accounts` from the left-side navigation menu.
3. On the Service Accounts page, click on `Create Service Account` at the top of the page.
4. Put the appropriate Service account name and Service account description and grant the service account `Storage Admin` access. Click Done.
2. Go to IAM and select **Service Accounts** from the left-side navigation menu.
3. On the Service Accounts page, click on **Create Service Account** at the top of the page.
4. Put the appropriate Service account name and Service account description and grant the service account **Storage Admin** access. **Click Done**.
5. On the Service Accounts page, click on the newly created service account.
6. On the newly created service account page, go to the `Keys` tab at the top of the page and click on `Add Key.` From the dropdown, select `Create New Key`.
7. Select the Key type as `JSON` and click on `Create`. The JSON key file for the service account will get downloaded.
8. We will be using the stringified JSON for our configuration. You can use the bash command `% cat <path_to_json_file> | jq '@json'` to generate stringified JSON.
6. On the newly created service account page, go to the **Keys** tab at the top of the page and click on **Add Key**. From the dropdown, select **Create New Key**.
7. Select the Key type as **JSON** and click on **Create**. The JSON key file for the service account will be downloaded.
8. We will be using the stringified JSON for our configuration. You can use the bash command `cat <path_to_json_file> | jq '@json'` to generate stringified JSON.
9. Edit Kestra storage configuration in the [Helm chart's values](https://github.com/kestra-io/helm-charts/blob/master/charts/kestra/values.yaml#L11).

::alert{type="info"}
*Note: If you want to use a Kubernetes service account configured as a workload identify, you don't need to provide anything for `serviceAccount` as it will be autodetected for the pod configuration if it's well configured.*
*Note: If you want to use a Kubernetes service account configured with Workload Identity, you don't need to provide anything for `serviceAccount`, as it will be autodetected for the pod configuration if it's well configured.*
::

```yaml
Expand All @@ -186,7 +186,7 @@ configuration:
"<stringified-json-file-contents>"
```

Disable the MinIO pod by changing `enabled` value in the `minio` section from `true` to `false` in the same file.
Disable the MinIO pod by changing the `enabled` value in the `minio` section from `true` to `false` in the same file.
```yaml
minio:
enabled: false
Expand Down Expand Up @@ -256,16 +256,16 @@ postgresql:

In this example:

-**Secrets**: You can configure sensitive values as secrets, either hardcoding them or referencing existing Kubernetes secrets.
-**Queue and Repository**: By default, these can use PostgreSQL or any other supported type. Uncomment the relevant lines to use them.
-**PostgreSQL Configuration**: Set the datasources section to provide details for connecting to a PostgreSQL database.
-**Disabling Services**: If you're using external services like CloudSQL or Google Cloud Storage, you can disable the built-in services (MinIO and PostgreSQL).
- **Secrets**: You can configure sensitive values as secrets, either hardcoding them or referencing existing Kubernetes secrets.
- **Queue and Repository**: By default, these can use PostgreSQL or any other supported type. Uncomment the relevant lines to use them.
- **PostgreSQL Configuration**: Set the datasources section to provide details for connecting to a PostgreSQL database.
- **Disabling Services**: If you're using external services like CloudSQL or Google Cloud Storage, you can disable the built-in services (MinIO and PostgreSQL).


Feel free to uncomment and modify these examples based on your setup needs. This provides flexibility while keeping your values.yaml well-structured.

## Next steps

This guide walked you through installing Kestra to [Google GKE](https://cloud.google.com/kubernetes-engine/docs) with CloudSQL as database and Google Cloud Storage as storage backend.
This guide walked you through installing Kestra to [Google GKE](https://cloud.google.com/kubernetes-engine/docs) with CloudSQL as the database and Google Cloud Storage as the storage backend.

Reach out via [Slack](/slack) if you encounter any issues or if you have any questions regarding deploying Kestra to production.
Reach out via [Slack](/slack) if you encounter any issues or have any questions regarding deploying Kestra to production.