Skip to content

Commit

Permalink
update: Flink getting started and integrate service (#237)
Browse files Browse the repository at this point in the history
  • Loading branch information
harshini-rangaswamy authored Jun 10, 2024
1 parent 98b7e36 commit 173d99f
Show file tree
Hide file tree
Showing 7 changed files with 152 additions and 82 deletions.
146 changes: 104 additions & 42 deletions docs/products/flink/get-started.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,124 @@
---
title: Aiven for Apache Flink® quickstart guide
title: Get started with Aiven for Apache Flink®
sidebar_label: Get started
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import ConsoleLabel from "@site/src/components/ConsoleIcons"

The first step in using Aiven for Apache Flink® is to create a service.
You can do this in the [Aiven web console](https://console.aiven.io/) or
with the [Aiven CLI](https://github.com/aiven/aiven-client).
Begin your experience with Aiven for Apache Flink® by setting up a service, configuring data integrations, and building streaming applications.

This quickstart section provides you with the information you need to
get started with Aiven for Apache Flink and build data pipelines to
stream to analyze your data. Learn how to set up and configure Aiven for
Flink, connect to your desired data sources, and start processing and
analyzing your data in real-time.
## Prerequisites

## Create an Aiven for Apache Flink® service
- Access to the [Aiven Console](https://console.aiven.io)
- [Aiven CLI](https://github.com/aiven/aiven-client)

1. Log in to the [Aiven web console](https://console.aiven.io/).
## Create a service

2. Follow
[these instructions](/docs/platform/howto/create_new_service) to create an Aiven for Apache Flink service.
<Tabs groupId="group1">
<TabItem value="1" label="Console" default>

1. Log in to [Aiven Console](https://console.aiven.io).
1. [Create an Aiven for Apache Flink® service](/docs/platform/howto/create_new_service).

Once the service is ready, the status changes to *Running*.
Depending on your selected cloud provider and region, this generally
takes a couple of minutes.
</TabItem>
<TabItem value="2" label="CLI">

## Create data service integration
1. Set up your authentication token by following
[these instructions](/docs/platform/howto/create_authentication_token).
1. Execute the following command:

Aiven for Apache Flink® allows you to build data pipelines that stream
across different services. It currently supports integration with Aiven
for Apache Kafka®, Aiven for PostgreSQL®, and Aiven for OpenSearch® as
sources and targets for Flink applications.
```bash
avn service create flink-demo \
--service-type flink \
--cloud google-europe-west3 \
--plan business-4 \
-c flink_version=1.19
```

To create your first data service integration:
Parameters:

1. Open your Aiven for Apache Flink.
2. In the service **Overview** screen, select **Get Started**.
3. Choose the service you wish to integrate: Aiven for Apache Kafka®,
Aiven for PostgreSQL®, or Aiven for OpenSearch®.
4. Select **Integrate** to complete the integration process.
- `flink-demo`: The `service_name` for your new service.
- `--service-type flink`: The type of service to be created, with `flink` indicating an Aiven for Apache Flink service.
- `--cloud google-europe-west3`: Identifies the cloud region where the service
will be hosted.
- `--plan business-4`: Refers to the subscription plan that dictates the resource
allocation for your service.
- `-c flink_version=1.19`: This configuration setting specifies the version of
Apache Flink that your service will run.

For detailed steps, see
[Create Apache Flink® data service integrations](howto/create-integration).
</TabItem>
</Tabs>

## Create Aiven for Apache Flink® application
## Configure data service integrations

An
[Aiven for Apache Flink® application](concepts/flink-applications) is an abstraction layer on top of Apache Flink SQL that
includes all the elements related to a Flink job to help build your data
processing pipeline.
<Tabs groupId="group1">
<TabItem value="1" label="Console" default>

Applications are the starting point for running an Apache Flink job
within the Aiven managed service and contains the definition of source
and sink tables, data processing logic, deployment parameters, and other
necessary metadata.
Aiven for Apache Flink® streamlines data processing by enabling integration with
various services. Currently, it supports integration with:

- **Aiven for Apache Kafka®** and **external Apache Kafka clusters**
- **Aiven for PostgreSQL®**
- **Aiven for OpenSearch®**
- **Google BigQuery®**

### Integration steps

1. Log in to [Aiven Console](https://console.aiven.io) and access your
Aiven for Apache Flink service.
1. On the <ConsoleLabel name="overview"/> page, scroll to **Data pipeline**.
1. Click **Add data source**.
1. Choose the service you wish to integrate.
1. Click **Integrate**.

:::tip
For detailed integration steps with specific services, see the [Integrate service](/docs/products/flink/howto/list-integrations).
:::

</TabItem>
<TabItem value="2" label="CLI">
You can set up your data service integrations using the Aiven CLI
with the following command:

```bash
avn service integration-create \
--project demo-sandbox \
--source-service kafka-demo \
--dest-service flink-demo \
--integration-type flink
```

Parameters:

- `--project demo-sandbox`: Defines the project for the integration.
- `--source-service kafka-demo`: Sets Aiven for Apache Kafka service
as the data source.
- `--dest-service flink-demo`: Sets Aiven for Apache Flink service as
the data destination.
- `--integration-type flink`: Determines the integration type, enabling data transfer
from Aiven for Apache Kafka to Aiven for Apache Flink.

</TabItem>
</Tabs>

## Create an Aiven for Apache Flink® application

An [Aiven for Apache Flink® application](concepts/flink-applications) act as containers
that encapsulate all aspects of a Flink job. This includes data source and sink
connections, as well as the data processing logic.

### Application types

- **SQL applications**: These applications are ideal for executing SQL queries. The
Aiven Console guides you through selecting source and sink tables, and setting up the
SQL queries to process your data. To learn how to create SQL applications, see
[Create an SQL application](/docs/products/flink/howto/create-sql-application).
- **JAR applications**:These applications allow you to deploy custom functionalities. The
Aiven Console enables you to upload and manage your JAR files and execute custom Flink
jobs that go beyond the standard SQL capabilities. To learn how to create
JAR applications, see [Create a JAR application](/docs/products/flink/howto/create-jar-application).

For information on creating Flink applications and integrating data
services, see
[Create an Aiven for Apache Flink® application](howto/create-flink-applications) and
[Create Apache Flink® data service integrations](howto/create-integration) sections.

## Next steps

Expand Down
68 changes: 41 additions & 27 deletions docs/products/flink/howto/create-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,49 @@
title: Create Apache Flink® data service integrations
---
import ConsoleLabel from "@site/src/components/ConsoleIcons"
import ConsoleIcon from "@site/src/components/ConsoleIcons"

With Apache Flink®, you can create streaming data pipelines across services. Aiven for Apache Flink® currently supports Aiven for Apache Kafka®, Aiven for PostgreSQL®, and Aiven for OpenSearch® as sources and targets for Flink applications.
With Aiven for Apache Flink®, you can create streaming data pipelines to connect various services.

To use Aiven for Apache Kafka, Aiven for PostgreSQL, or Aiven for
OpenSearch as a source or target for a Flink application, you will need
to integrate the related service with Aiven for Apache Flink.
Aiven for Apache Flink® enables integration with internal and external services,
such as Aiven managed services, Google BigQuery®, and external Apache Kafka and
PostgreSQL instances. This flexibility allows you to build data processing pipelines
using a wide range of data sources and sinks.

## Create data service integration

You can create Aiven for Apache Flink® data service integrations
via the [Aiven Console](https://console.aiven.io/) by following these
steps:

1. Go to the Aiven for Apache Flink® service page.

2. If you are setting up the first integration for the selected Aiven
for Apache Flink service, click **Get Started** in the service
**Overview** screen.

![Image of the Aiven for Apache Flink Overview page with focus on the Get Started Icon](/images/content/products/flink/integrations-get-started.png)

3. To configure the data flow with Apache Flink®, click the Aiven for
Apache Kafka®, Aiven for PostgreSQL®, or Aiven for OpenSearch®
service that you wish to integrate. Click <ConsoleLabel name="integrations"/>.
to complete the integration process.

![Image of the Aiven for Apache Flink Integration page showing an Aiven for Apache Kafka® and an Aiven for PostgreSQL® services](/images/content/products/flink/integrations-select-services.png)

4. You can include additional integrations by clicking <ConsoleLabel name="plus"/>
in the **Data Flow** section.

![Image of the Aiven for Apache Flink Integration page showing an existing Aiven for Apache Kafka integration and the + icon to add additional integrations](/images/content/products/flink/integrations-add.png)
Create Aiven for Apache Flink® data service integrations using the
[Aiven Console](https://console.aiven.io/). Follow these steps to integrate
either internal and external services.

### Integrate Aiven services

1. Log in to [Aiven Console](https://console.aiven.io) and access your
Aiven for Apache Flink service.
1. If this is your first integration for the selected Aiven for Apache Flink service,
on the <ConsoleLabel name="overview"/> page, scroll to **Data pipeline**.
1. Click **Add data source** to initiate the integration setup.
1. On the **Data service integrations** page, under **Create service integration** tab,
choose the Aiven service to integrate: Aiven for Apache Kafka®,
Aiven for PostgreSQL®, or Aiven for OpenSearch®.
1. Click **Integrate**.
1. To include additional integrations, click <ConsoleIcon name="plus"/> in the
**Data pipeline** section.

### Integrate external services

1. On the **Data service integrations** page, click the
**Create external integration endpoint** tab.
1. Select the type of external service to integrate: Google BigQuery®,
External Apache Kafka, or External PostgreSQL.
1. If no external endpoints are available, go back to the **Projects** page.
1. Click <ConsoleLabel name="integrationendpoints"/> to add and configure external
integration points.
1. Once the integration endpoint is added, return to your Aiven for Apache Flink service
and the **Data pipeline** section.
1. Click **Add data source**.
1. On the **Data service integrations** page, under
**Create external integration endpoint** tab, select the checkbox next to the
external data service type and choose the integration endpoint just created.
1. To include additional integrations, click <ConsoleIcon name="plus"/> in the
**Data pipeline** section.
11 changes: 0 additions & 11 deletions docs/products/flink/howto/list-get-started.md

This file was deleted.

4 changes: 4 additions & 0 deletions docs/products/flink/list-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,7 @@ The topics in this section provide an understanding of Aiven for Apache
Flink® and its features. It also covers the managed service features,
plans, and pricing options available, as well as any limitations to be
aware of when using the service.

import DocCardList from '@theme/DocCardList';

<DocCardList />
3 changes: 1 addition & 2 deletions sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1009,7 +1009,6 @@ const sidebars: SidebarsConfig = {
id: 'products/flink/howto',
},
items: [
'products/flink/howto/list-get-started',
{
type: 'category',
label: 'Integrate service',
Expand Down Expand Up @@ -1046,7 +1045,6 @@ const sidebars: SidebarsConfig = {
id: 'products/flink/howto/list-flink-tables',
},
items: [
'products/flink/howto/manage-flink-tables',
{
type: 'category',
label: 'Create tables with data source',
Expand All @@ -1064,6 +1062,7 @@ const sidebars: SidebarsConfig = {
'products/flink/howto/datagen-connector',
],
},
'products/flink/howto/manage-flink-tables',
],
},
'products/flink/howto/list-manage-cluster',
Expand Down
2 changes: 2 additions & 0 deletions static/_redirects
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,5 @@
/tools/terraform/howto/vnet-peering-azure https://github.com/Aiven-Open/terraform-example-projects/blob/main/virtual_private_networking/azure_vnet_peering
/tools/terraform/howto/vpc-peering-gcp https://github.com/Aiven-Open/terraform-example-projects/tree/main/virtual_private_networking/google_vpc_peering
/tools/terraform/list-vpc-terraform https://github.com/Aiven-Open/terraform-example-projects/tree/main/virtual_private_networking

/products/flink/howto/list-get-started https://aiven.io/docs/products/flink/get-started
Binary file not shown.

0 comments on commit 173d99f

Please sign in to comment.