forked from apache/seatunnel
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Feature][Flink-SQL-connector] add flink sql connector kafka and docs (…
…apache#1878) * [Feature][Flink-SQL-connector] add flink sql connector kafka and docs * address review comment Co-authored-by: ruanwenjun <wenjun@apache.org>
- Loading branch information
1 parent
91ddc2e
commit 6cade31
Showing
13 changed files
with
136 additions
and
25 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Flink SQL Kafka Connector | ||
|
|
||
| ## Description | ||
|
|
||
| With kafka connector, we can read data from kafka and write data to kafka using Flink SQL. Refer to the [Kafka connector](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/kafka/) for more details. | ||
|
|
||
|
|
||
| ## Usage | ||
| Let us have a brief example to show how to use the connector from end to end. | ||
|
|
||
| ### 1. kafka prepare | ||
| Please refer to the [Kafka QuickStart](https://kafka.apache.org/quickstart) to prepare kafka environment and produce data like following: | ||
|
|
||
| ```bash | ||
| $ bin/kafka-console-producer.sh --topic <topic-name> --bootstrap-server localhost:9092 | ||
| ``` | ||
|
|
||
| After executing the command, we will come to the interactive mode. Print the following message to send data to kafka. | ||
| ```bash | ||
| {"id":1,"name":"abc"} | ||
| >{"id":2,"name":"def"} | ||
| >{"id":3,"name":"dfs"} | ||
| >{"id":4,"name":"eret"} | ||
| >{"id":5,"name":"yui"} | ||
| ``` | ||
|
|
||
| ### 2. prepare seatunnel configuration | ||
| Here is a simple example of seatunnel configuration. | ||
| ```sql | ||
| SET table.dml-sync = true; | ||
|
|
||
| CREATE TABLE events ( | ||
| id INT, | ||
| name STRING | ||
| ) WITH ( | ||
| 'connector' = 'kafka', | ||
| 'topic'='<topic-name>', | ||
| 'properties.bootstrap.servers' = 'localhost:9092', | ||
| 'properties.group.id' = 'testGroup', | ||
| 'scan.startup.mode' = 'earliest-offset', | ||
| 'format' = 'json' | ||
| ); | ||
|
|
||
| CREATE TABLE print_table ( | ||
| id INT, | ||
| name STRING | ||
| ) WITH ( | ||
| 'connector' = 'print', | ||
| 'sink.parallelism' = '1' | ||
| ); | ||
|
|
||
| INSERT INTO print_table SELECT * FROM events; | ||
| ``` | ||
|
|
||
| ### 3. start flink local cluster | ||
| ```bash | ||
| $ ${FLINK_HOME}/bin/start-cluster.sh | ||
| ``` | ||
|
|
||
| ### 4. start Flink SQL job | ||
| Execute the following command in seatunnel home path to start the Flink SQL job. | ||
| ```bash | ||
| $ bin/start-seatunnel-sql.sh -c config/kafka.sql.conf | ||
| ``` | ||
|
|
||
| ### 5. verify result | ||
| After the job submitted, we can see the data printing by connector 'print' in taskmanager's log . | ||
| ```text | ||
| +I[1, abc] | ||
| +I[2, def] | ||
| +I[3, dfs] | ||
| +I[4, eret] | ||
| +I[5, yui] | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
36 changes: 36 additions & 0 deletions
36
seatunnel-connectors/seatunnel-connectors-flink-sql/flink-sql-connector-kafka/pom.xml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <parent> | ||
| <artifactId>seatunnel-connectors-flink-sql</artifactId> | ||
| <groupId>org.apache.seatunnel</groupId> | ||
| <version>${revision}</version> | ||
| </parent> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <artifactId>flink-sql-connector-kafka</artifactId> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.flink</groupId> | ||
| <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId> | ||
| </dependency> | ||
|
|
||
| </dependencies> | ||
|
|
||
| </project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters