Skip to content

Commit

Permalink
improve documentation around logging in kubernetes (#8072)
Browse files Browse the repository at this point in the history
* improve documentation around logging in kubernetes

* add placeholders in env vars

* add extra clause
  • Loading branch information
lmossman authored Nov 19, 2021
1 parent dc04654 commit 93c63a3
Show file tree
Hide file tree
Showing 2 changed files with 34 additions and 24 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,25 @@ static CloudLogs createCloudLogClient(final LogConfigs configs) {
throw new RuntimeException("Error no cloud credentials configured..");
}

/**
* Logs are configured to go to a Minio instance if the S3MinioEndpoint configuration specifically is set, along with the
* other default S3 logging configurations.
*
* @param configs contains the environment variables to check
* @return boolean indicating if logs are configured to be sent to minio
*/
private static boolean hasMinioConfiguration(final LogConfigs configs) {
return !configs.getS3LogBucket().isBlank() && !configs.getAwsAccessKey().isBlank()
&& !configs.getAwsSecretAccessKey().isBlank() && !configs.getS3MinioEndpoint().isBlank();
}

/**
* Logs are configured to go to S3 if the S3LogBucketRegion configuration specifically is set, along with the
* other default S3 logging configurations.
*
* @param configs contains the configuration values to check
* @return boolean indicating if logs are configured to be sent to S3
*/
private static boolean hasS3Configuration(final LogConfigs configs) {
return !configs.getAwsAccessKey().isBlank() &&
!configs.getAwsSecretAccessKey().isBlank() &&
Expand Down
44 changes: 20 additions & 24 deletions docs/deploying-airbyte/on-kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,42 +46,39 @@ Configure `kubectl` to connect to your cluster by using `kubectl use-context my-

### Configure Logs

#### Default Configuration
Both `dev` and `stable` versions of Airbyte include a stand-alone `Minio` deployment. Airbyte publishes logs to this `Minio` deployment by default. This means Airbyte comes as a **self-contained Kubernetes deployment - no other configuration is required**.

Airbyte currently supports logging to `Minio`, `S3` or `GCS`. The following instructions are for users wishing to log to their own `Minio` layer, `S3` bucket or `GCS` bucket.
So if you just want logs to be sent to the local `Minio` deployment, you do not need to change the values of any environment variables from what is currently on master.

#### Custom Configuration

Alternatively, if you want logs to be sent to a custom location, Airbyte currently supports logging to `Minio`, `S3` or `GCS`. The following instructions are for users wishing to log to their own `Minio` layer, `S3` bucket or `GCS` bucket.

The provided credentials require both read and write permissions. The logger attempts to create the log bucket if it does not exist.

#### Configuring Custom Minio Log Location
##### Configuring Custom Minio Log Location

Replace the following variables in the `.env` file in the `kube/overlays/stable` directory:
To write to a custom minio log location, replace the following variables in the `.env` file in the `kube/overlays/stable` directory:

```text
# The Minio bucket to write logs in.
S3_LOG_BUCKET=
# Minio Access Key.
AWS_ACCESS_KEY_ID=
# Minio Secret Key.
AWS_SECRET_ACCESS_KEY=
# Endpoint where Minio is deployed at.
S3_MINIO_ENDPOINT=
S3_LOG_BUCKET=<your_minio_bucket_to_write_logs_in>
AWS_ACCESS_KEY_ID=<your_minio_access_key>
AWS_SECRET_ACCESS_KEY=<your_minio_secret_key>
S3_MINIO_ENDPOINT=<endpoint_where_minio_is_deployed_at>
```

The `S3_PATH_STYLE_ACCESS` variable should remain `true`. The `S3_LOG_BUCKET_REGION` variable should remain empty.

#### Configuring Custom S3 Log Location
##### Configuring Custom S3 Log Location

Replace the following variables in the `.env` file in the `kube/overlays/stable` directory:
To write to a custom S3 log location, replace the following variables in the `.env` file in the `kube/overlays/stable` directory:

```text
# The S3 bucket to write logs in.
S3_LOG_BUCKET=
# The S3 bucket region.
S3_LOG_BUCKET_REGION=
# Aws Access Key Id.
AWS_ACCESS_KEY_ID=
# Aws Secret Access Key
AWS_SECRET_ACCESS_KEY=
S3_LOG_BUCKET=<your_s3_bucket_to_write_logs_in>
S3_LOG_BUCKET_REGION=<your_s3_bucket_region>
AWS_ACCESS_KEY_ID=<your_aws_access_key_id>
AWS_SECRET_ACCESS_KEY=<your_aws_secret_access_key>
# Set this to empty.
S3_MINIO_ENDPOINT=
# Set this to empty.
Expand All @@ -90,7 +87,7 @@ S3_PATH_STYLE_ACCESS=

See [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) for instructions on creating an S3 bucket and [here](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) for instructions on creating AWS credentials.

#### Configuring Custom GCS Log Location
##### Configuring Custom GCS Log Location

Create the GCP service account with read/write permission to the GCS log bucket.

Expand All @@ -116,8 +113,7 @@ data:
3\) Replace the following variables in the `.env` file in the `kube/overlays/stable` directory:

```text
# The GCS bucket to write logs in.
GCP_STORAGE_BUCKET=
GCP_STORAGE_BUCKET=<your_GCS_bucket_to_write_logs_in>
# The path the GCS creds are written to. Unless you know what you are doing, use the below default value.
GOOGLE_APPLICATION_CREDENTIALS=/secrets/gcs-log-creds/gcp.json
```
Expand Down

0 comments on commit 93c63a3

Please sign in to comment.