Skip to content

refactor(clickpipes): reverse private link for postgres #3696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 18, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ that will be accessible in a ClickPipes VPC.
It's a preferred choice for:

- Any on-premise Kafka setup that requires private DNS support
- Cross-region connectivity for Postgres CDC
- [Cross-region connectivity for Postgres CDC](/knowledgebase/aws-privatelink-setup-for-clickpipes)
- Cross-region connectivity for MSK cluster. Please reach out to the ClickHouse support team for assistance.

See the [getting started](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) guide for more details.
Expand Down Expand Up @@ -113,8 +113,10 @@ can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-r
5. Select any of existing reverse private endpoints or create a new one.

:::info
For RDS, the process to create the VPC Endpoint service is a bit more involved and
If cross-region access is required for RDS, you need to create a VPC endpoint service and
[this guide should provide](/knowledgebase/aws-privatelink-setup-for-clickpipes) a good starting point to set it up.

For same-region access, creating a VPC Resource is the recommended approach.
:::

<Image img={cp_rpe_step1} alt="Select reverse private endpoint" size="lg" border/>
Expand Down
27 changes: 15 additions & 12 deletions docs/integrations/data-ingestion/clickpipes/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import S3svg from '@site/static/images/integrations/logos/amazon_s3_logo.svg';
import Amazonkinesis from '@site/static/images/integrations/logos/amazon_kinesis_logo.svg';
import Gcssvg from '@site/static/images/integrations/logos/gcs.svg';
import Postgressvg from '@site/static/images/integrations/logos/postgresql.svg';
import Mysqlsvg from '@site/static/images/integrations/logos/mysql.svg';
import redpanda_logo from '@site/static/images/integrations/logos/logo_redpanda.png';
import clickpipes_stack from '@site/static/images/integrations/data-ingestion/clickpipes/clickpipes_stack.png';
import cp_custom_role from '@site/static/images/integrations/data-ingestion/clickpipes/cp_custom_role.png';
Expand All @@ -29,18 +30,20 @@ import Image from '@theme/IdealImage';

## Supported Data Sources {#supported-data-sources}

| Name |Logo|Type| Status | Description |
|----------------------|----|----|-----------------|------------------------------------------------------------------------------------------------------|
| Apache Kafka |<Kafkasvg class="image" alt="Apache Kafka logo" style={{width: '3rem', 'height': '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Apache Kafka into ClickHouse Cloud. |
| Confluent Cloud |<Confluentsvg class="image" alt="Confluent Cloud logo" style={{width: '3rem'}}/>|Streaming| Stable | Unlock the combined power of Confluent and ClickHouse Cloud through our direct integration. |
| Redpanda |<Image img={redpanda_logo} size="logo" alt="Redpanda logo"/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Redpanda into ClickHouse Cloud. |
| AWS MSK |<Msksvg class="image" alt="AWS MSK logo" style={{width: '3rem', 'height': '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from AWS MSK into ClickHouse Cloud. |
| Azure Event Hubs |<Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. |
| WarpStream |<Warpstreamsvg class="image" alt="WarpStream logo" style={{width: '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from WarpStream into ClickHouse Cloud. |
| Amazon S3 |<S3svg class="image" alt="Amazon S3 logo" style={{width: '3rem', height: 'auto'}}/>|Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
| Google Cloud Storage |<Gcssvg class="image" alt="Google Cloud Storage logo" style={{width: '3rem', height: 'auto'}}/>|Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
| Amazon Kinesis |<Amazonkinesis class="image" alt="Amazon Kenesis logo" style={{width: '3rem', height: 'auto'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse cloud. |
| Postgres |<Postgressvg class="image" alt="Postgres logo" style={{width: '3rem', height: 'auto'}}/>|DBMS| Public Beta | Configure ClickPipes and start ingesting data from Postgres into ClickHouse Cloud. |
| Name | Logo |Type| Status | Description |
|----------------------|--------------------------------------------------------------------------------------------------|----|------------------|------------------------------------------------------------------------------------------------------|
| Apache Kafka | <Kafkasvg class="image" alt="Apache Kafka logo" style={{width: '3rem', 'height': '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Apache Kafka into ClickHouse Cloud. |
| Confluent Cloud | <Confluentsvg class="image" alt="Confluent Cloud logo" style={{width: '3rem'}}/> |Streaming| Stable | Unlock the combined power of Confluent and ClickHouse Cloud through our direct integration. |
| Redpanda | <Image img={redpanda_logo} size="logo" alt="Redpanda logo"/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Redpanda into ClickHouse Cloud. |
| AWS MSK | <Msksvg class="image" alt="AWS MSK logo" style={{width: '3rem', 'height': '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from AWS MSK into ClickHouse Cloud. |
| Azure Event Hubs | <Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. |
| WarpStream | <Warpstreamsvg class="image" alt="WarpStream logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from WarpStream into ClickHouse Cloud. |
| Amazon S3 | <S3svg class="image" alt="Amazon S3 logo" style={{width: '3rem', height: 'auto'}}/> |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
| Google Cloud Storage | <Gcssvg class="image" alt="Google Cloud Storage logo" style={{width: '3rem', height: 'auto'}}/> |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
| Amazon Kinesis | <Amazonkinesis class="image" alt="Amazon Kenesis logo" style={{width: '3rem', height: 'auto'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse cloud. |
| Postgres | <Postgressvg class="image" alt="Postgres logo" style={{width: '3rem', height: 'auto'}}/> |DBMS| Public Beta | Configure ClickPipes and start ingesting data from Postgres into ClickHouse Cloud. |
| MySQL | <Mysqlsvg class="image" alt="MySQL logo" style={{width: '3rem', height: 'auto'}}/> |DBMS| Private Beta | Configure ClickPipes and start ingesting data from MySQL into ClickHouse Cloud. |


More connectors will get added to ClickPipes, you can find out more by [contacting us](https://clickhouse.com/company/contact?loc=clickpipes).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,12 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have

<Image img={postgres_connection_details} alt="Fill in connection details" size="lg" border/>

#### (Optional) Setting up AWS Private Link {#optional-setting-up-aws-private-link}

You can use AWS Private Link to connect to your source Postgres database if it is hosted on AWS. This is useful if you
want to keep your data transfer private.
You can follow the [setup guide to set up the connection](/integrations/clickpipes/aws-privatelink).

#### (Optional) Setting up SSH Tunneling {#optional-setting-up-ssh-tunneling}

You can specify SSH tunneling details if your source Postgres database is not publicly accessible.
Expand Down
11 changes: 6 additions & 5 deletions knowledgebase/aws-privatelink-setup-for-clickpipes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,13 @@ keywords: ['AWS PrivateLink', 'Private RDS', 'ClickPipes']

## AWS PrivateLink setup to expose private RDS for ClickPipes {#aws-privatelink-setup-to-expose-private-rds-for-clickpipes}

Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes.
Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes, also supporting cross-region access.

## Requirements {#requirements}

The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1, additionally the
ClickHouse Instance must be in the same region.
:::info
This uses AWS Endpoint Service and is mainly suitable for Cross-Region Access. If you require same-region access,
[creating a VPC Resource](/integrations/clickpipes/aws-privatelink#vpc-resource) is
recommended instead.
:::

## Private link creation {#private-link-creation}

Expand Down
1 change: 1 addition & 0 deletions scripts/aspell-dict-file.txt
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ ConfluentSVG
GCSSVG
KafkaSVG
MskSVG
Mysqlsvg
PostgresSVG
WarpStreamSVG
clickpipes
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.