Skip to content

Commit

Permalink
arch diagrams
Browse files Browse the repository at this point in the history
  • Loading branch information
@wojcik-dorota
wojcik-dorota committed Oct 4, 2024
1 parent fd9a97c commit be7973e
Show file tree
Hide file tree
Showing 6 changed files with 69 additions and 98 deletions.
29 changes: 18 additions & 11 deletions docs/platform/concepts/byoc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,30 +110,38 @@ traffic through a proxy for additional security. To accomplish this, Aiven
utilizes a bastion host (**Bastion node**) physically separated from the Aiven services
you deploy. The service VMs reside in a privately addressed subnet (**Private subnet**)
and are accessed by the Aiven management plane via the bastion. They are not
accessible through the Internet.
accessible through the internet.

:::note
Although the bastion host and the service nodes reside in the VPC under
your management (**BYOC VPC**), they are not accessible (for example, via SSH) to anyone
outside Aiven.

The bastion and workload nodes require outbound access to the Internet
The bastion and workload nodes require outbound access to the internet
to work properly (supporting HA signaling to the Aiven management node and RPM download
from Aiven repositories).
:::

The private subnet is where your object storage (two S3 buckets) reside. This storage is
used for [service backups](/docs/platform/concepts/byoc#byoc-service-backups) and as cold
[storage for your service's data](/docs/platform/howto/byoc/store-data).

</TabItem>
<TabItem value="2" label="AWS public">

<img src={byocAwsPublic} className="centered" alt="BYOC AWS public architecture" width="100%" />

In the AWS public deployment model, a Virtual Private Cloud (**BYOC VPC**) for your Aiven
services is created within a particular cloud region in your remote cloud account.
Aiven accesses this VPC through an Internet gateway. Service VMs reside in a publicly
Aiven accesses this VPC through an internet gateway. Service VMs reside in a publicly
addressed subnet (**Public subnet**), and Aiven services can be accessed
through the public Internet: the Aiven control plane connects to the nodes
through the public internet: the Aiven control plane connects to the nodes
using the public address, and the Aiven management plane can access the service VMs
directly.

The public subnet is where your object storage (two S3 buckets) reside. This storage is
used for [service backups](/docs/platform/concepts/byoc#byoc-service-backups) and as cold
[storage for your service's data](/docs/platform/howto/byoc/store-data).
</TabItem>
<TabItem value="3" label="Google Cloud private">

Expand All @@ -151,14 +159,14 @@ traffic through a proxy for additional security. To accomplish this, Aiven
utilizes a bastion host (**Bastion note**) physically separated from the Aiven services
you deploy. The service VMs reside in a privately addressed subnet (**Private subnet**)
and are accessed by the Aiven management plane via the bastion. They are not
accessible through the Internet.
accessible through the internet.

:::note
Although the bastion host and the service nodes reside in the VPC under
your management (**BYOC VPC**), they are not accessible (for example, via SSH) to anyone
outside Aiven.

The bastion and workload nodes require outbound access to the Internet
The bastion and workload nodes require outbound access to the internet
to work properly (supporting HA signaling to the Aiven management node and RPM download
from Aiven repositories).
:::
Expand All @@ -170,9 +178,9 @@ from Aiven repositories).

In the Google Cloud public deployment model, a Virtual Private Cloud (**Workload VPC**)
for your Aiven services is created within a particular cloud region in your remote cloud
account. Aiven accesses this VPC through an Internet gateway. Service VMs reside in a
account. Aiven accesses this VPC through an internet gateway. Service VMs reside in a
publicly addressed subnet (**Public subnet**), and Aiven services can be accessed
through the public Internet: the Aiven control plane connects to the nodes
through the public internet: the Aiven control plane connects to the nodes
using the public address, and the Aiven management plane can access the service VMs
directly.
</TabItem>
Expand All @@ -188,7 +196,7 @@ Depending on the service used, Aiven takes regular backups to enable forking, po
time recovery (PITR), and disaster recovery.

- Backups of services hosted using AWS BYOC reside in object storage in your own cloud
account.
account.
- Backups of BYOC services hosted using a cloud provider other than AWS reside in Aiven-owned
storage by default. It's still possible to store such backups in your own cloud account,
provided Aiven gets read-write permissions to access the object storage in your cloud
Expand All @@ -199,8 +207,7 @@ account.
- You are responsible for managing object storage configuration.
:::

To learn more about how data and backups are stored in BYOC, see
[Storing data in custom clouds](/docs/platform/howto/byoc/store-data).
Learn more about [storing data in custom clouds](/docs/platform/howto/byoc/store-data).

## Dev tools for BYOC

Expand Down
16 changes: 8 additions & 8 deletions docs/platform/howto/byoc/create-custom-cloud/create-aws-custom-cloud.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -478,18 +478,18 @@ In the **Create custom cloud** wizard:
cannot change the BYOC VPC CIDR block after your custom
cloud is created.

- BYOC remote storage
- Remote storage (BYOC-hosted)

By default, data is stored in your own cloud account's object storage using one S3
bucket per custom cloud.
By default, the following data is stored in object storage in your own cloud account:

- [Tiered storage](/docs/platform/howto/byoc/store-data) (with object storage as
a tier for historical or rarely queried data)
- Backups
- Cold data (learn more about the
[BYOC tiered storage](/docs/platform/howto/byoc/store-data#byoc-tiered-storage))
- Service backups

:::note
Permissions for S3 bucket management will be included in the Terraform
infrastructure template to be generated upon completing this step.
- Data is stored in your object storage using one S3 bucket per custom cloud.
- Permissions for S3 bucket management will be included in the Terraform
infrastructure template to be generated upon completing this step.
:::

1. Click **Next**.
Expand Down
19 changes: 6 additions & 13 deletions docs/platform/howto/byoc/create-custom-cloud/create-google-custom-cloud.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ In the **Create custom cloud** wizard:

1. Click **Next**.

1. Set up deployment <!-- and storage --> details:
1. Set up deployment and storage details:

- [Deployment model](/docs/platform/concepts/byoc#byoc-deployment)

Expand Down Expand Up @@ -158,21 +158,14 @@ In the **Create custom cloud** wizard:
cannot change the BYOC VPC CIDR block after your custom
cloud is created.

<!--
- BYOC remote storage
- Remote storage (Aiven-hosted)

By default, data is stored in your own cloud account's object storage using one S3
bucket per service.
By default, the following data is stored in the Aiven-owned cloud:

- [Tiered storage](/docs/platform/howto/byoc/store-data) (with object storage as
a tier for historical or rarely queried data)
- Backups
- Cold data (rarely accessed, static, or archived data grouped as a tier using
the tiered storage capability)
- Service backups

:::note
Permissions for S3 bucket management will be included in the Terraform
infrastructure template to be generated upon completing this step.
:::
-->
1. Click **Next**.

Your infrastructure Terraform template gets generated based on your inputs. You can
Expand Down
103 changes: 37 additions & 66 deletions docs/platform/howto/byoc/store-data.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,41 @@ keywords: [bring your own cloud, byoc, custom cloud, BYOC cloud, object storage,

import ConsoleLabel from "@site/src/components/ConsoleIcons";

Bring your own cloud (BYOC) allows you to use object storage in your own remote cloud account to store [cold data](/docs/platform/howto/byoc/store-data#tiered-storage) and [backups](/docs/platform/howto/byoc/store-data#user-owned-backups).
Depending on your cloud provider, data in your custom cloud and service backups can be stored either in Aiven-owned cloud or in your own cloud account.

## Tiered storage
## BYOC tiered storage

You can store data hosted in your custom cloud using tiered storage, a data allocation
mechanism for improved efficiency and cost optimization. When enabled, tiered storage
allows moving data automatically between hot storage (for frequently accessed, critical,
and often updated data) and cold storage (for rarely accessed, static, or archived data).
:::important
BYOC tiered storage is only supported in AWS custom clouds for
[Aiven for Apache Kafka](/docs/products/kafka/howto/kafka-tiered-storage-get-started) and
[Aiven for ClickHouse](/docs/products/clickhouse/concepts/clickhouse-tiered-storage).
:::

To store data, AWS [BYOC](/docs/platform/concepts/byoc) environments use tiered storage, a
data allocation mechanism for improved efficiency and cost optimization of data management.
When enabled, tiered storage allows moving data automatically between hot storage (for
frequently accessed, critical, and often updated data) and cold storage (for rarely
accessed, static, or archived data).

Cold storage for BYOC-hosted services uses object storage in your own remote cloud account.
For purposes of the cold storage:
Cold data of AWS-BYOC-hosted services is stored in object storage in your own AWS cloud
account. One S3 bucket is created per custom cloud.

- In AWS BYOC, one S3 bucket is created per custom cloud.
- In Google Cloud BYOC, one S3 bucket is created per BYOC-hosted service.
:::note

To use tiered storage in a BYOC-hosted service, tiered storage needs to be enabled both
- Non-BYOC services with Aiven-owned tiered storage cannot be migrated to BYOC.
- Tiered storage enabled on non-BYOC services doesn't allow to store cold data in your
own cloud account.

:::

To use tiered storage in an AWS-BYOC-hosted service, tiered storage needs to be enabled both
[in your custom cloud](/docs/platform/howto/byoc/store-data#enable-in-a-custom-cloud) and
[in the BYOC-hosted service](/docs/platform/howto/byoc/store-data#enable-on-a-service).

### Enable in a custom cloud

- Each custom cloud you create has tiered storage enabled by default.
- For existing custom clouds created in the past with no tiered storage support,
- **New AWS custom clouds** have tiered storage enabled by default.
- **Existing AWS custom clouds** created in the past with no tiered storage support,
[contact the Aiven support team](mailto:support@aiven.io) to request enabling tiered
storage.

Expand All @@ -40,7 +52,7 @@ You cannot deactivate tiered storage on your custom cloud once it's activated.

#### Prerequisites

- At least one [custom cloud](/docs/platform/howto/byoc/create-custom-cloud)
- At least one AWS [custom cloud](/docs/platform/howto/byoc/create-custom-cloud)
- At least one [Aiven-manged service](/docs/platform/howto/create_new_service), either
Aiven for Apache Kafka® or Aiven for ClickHouse®, hosted in a custom cloud

Expand All @@ -49,66 +61,25 @@ You cannot deactivate tiered storage on your custom cloud once it's activated.
[migrate to a custom cloud](/docs/platform/howto/byoc/manage-byoc-service#migrate-an-existing-service-to-a-custom-cloud).
:::

#### Limitations

- BYOC supports tiered storage for the following service types:
- [Aiven for Apache Kafka](/docs/products/kafka/howto/kafka-tiered-storage-get-started)
- [Aiven for ClickHouse](/docs/products/clickhouse/concepts/clickhouse-tiered-storage)
- Non-BYOC services with Aiven-owned tiered storage enabled cannot be migrated to BYOC,
and the tiered storage mechanism they use cannot be changed to use object storage in
your own remote cloud account as cold storage.

#### Activate tiered storage

You can activate tiered storage for A BYOC service either during service creation or on
an existing service.

##### Activate on a new service
- [Enable for Aiven for Apache Kafka](/docs/products/kafka/howto/enable-kafka-tiered-storage)
- [Enable for Aiven for Clickhouse](/docs/products/clickhouse/howto/enable-tiered-storage)

1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your organization.
1. Click **Admin** in the top navigation, and click <ConsoleLabel name="bringyourowncloud"/>
in the sidebar.
1. Select a custom cloud where to activate tiered storage, and go to the **Tiered storage**
tab.
1. Click **Activate tiered storage**, use the toggle for enabling tiered storage, and click
**Next**.
## BYOC service backups

Now, the updated infrastructure Terraform template and variables file are generated with
the new tiered storage configuration.
Aiven takes [regular service backups](/docs/platform/concepts/service_backups), which are
encrypted using Aiven-managed keys.

1. Copy or download the template and re-deploy it in your remote cloud account using the
variables provided in the variables file.
Backups of BYOC-hosted services are stored as follows:

##### Activate on an existing service

1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your organization.
1. Click **Admin** in the top navigation, and click <ConsoleLabel name="bringyourowncloud"/>
in the sidebar.
1. Select a custom cloud where to activate tiered storage, and go to the **Tiered storage**
tab.
1. Click **Activate tiered storage**, use the toggle for enabling tiered storage, and click
**Next**.

Now, the updated infrastructure Terraform template and variables file are generated with
the new tiered storage configuration.

1. Copy or download the template and re-deploy it in your remote cloud account using the
variables provided in the variables file.

## User-owned backups

By default, data backups of BYOC-hosted services are stored in object storage in your own
remote cloud account. For purposes of data backups:

- In AWS BYOC, one S3 bucket is created per custom cloud.
- In Google Cloud BYOC, one S3 bucket is created per BYOC-hosted service.
- **AWS BYOC**: User-owned backups stored in object storage in your own AWS cloud account.
One S3 bucket is created per custom cloud.
- **Google Cloud, Azure, or OCI BYOC**: Aiven-owned backups stored in Aiven-managed object
storage.

## Related pages

- [Enable tiered storage for Aiven for Apache Kafka](/docs/products/kafka/howto/enable-kafka-tiered-storage).
- [Enable tiered storage for Aiven for ClickHouse](/docs/products/kafka/howto/enable-kafka-tiered-storage).
- [About bring your own cloud](/docs/platform/concepts/byoc)
- [Bring your own cloud networking and security](/docs/platform/howto/byoc/networking-security)
- [Enable bring your own cloud (BYOC)](/docs/platform/howto/byoc/enable-byoc)
- [Create a custom cloud in Aiven](/docs/platform/howto/byoc/create-custom-cloud)
- [Download an infrastructure template and a variables file](/docs/platform/howto/byoc/download-infrastructure-template)
- [Bring your own cloud networking and security](/docs/platform/howto/byoc/networking-security)
Binary file modified static/images/content/figma/byoc-aws-private.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified static/images/content/figma/byoc-aws-public.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit be7973e

Please sign in to comment.