Skip to content

Commit 2e57ab1

Browse files
authored
Merge pull request GoogleCloudPlatform#504 from rangadi/kafka_release_notes
Increase KafkaIO version to 0.2.0
2 parents 8fd6661 + 40d174f commit 2e57ab1

File tree

2 files changed

+40
-1
lines changed

2 files changed

+40
-1
lines changed

contrib/kafka/README.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# KafkaIO : Dataflow Unbounded Source and Sink for Kafka Topics
2+
3+
KafkaIO provides unbounded source and sink for [Kafka](http://kafka.apache.org/)
4+
topics. Kafka versions 0.9 and above are supported.
5+
6+
## Basic Usage
7+
8+
* Read from a topic with 8 byte long keys and string values:
9+
```java
10+
PCollection<KV<Long, String>> kafkaRecords =
11+
pipeline
12+
.apply(KafkaIO.read()
13+
.withBootstrapServers("broker_1:9092,broker_2:9092")
14+
.withTopics(ImmutableList.of("topic_a"))
15+
.withKeyCoder(BigEndianLongCoder.of())
16+
.withValueCoder(StringUtf8Coder.of())
17+
.withoutMetadata()
18+
);
19+
```
20+
21+
* Write the same PCollection to a Kafka topic:
22+
```java
23+
kafkaRecords.apply(KafkaIO.write()
24+
.withBootstrapServers("broker_1:9092,broker_2:9092")
25+
.withTopic("results")
26+
.withKeyCoder(BigEndianLongCoder.of())
27+
.withValueCoder(StringUtf8Coder.of())
28+
);
29+
```
30+
31+
Please see JavaDoc for KafkaIO in
32+
[KafkaIO.java](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/contrib/kafka/src/main/java/com/google/cloud/dataflow/contrib/kafka/KafkaIO.java#L100)
33+
for complete documentation and a more descriptive usage example.
34+
35+
## Release Notes
36+
* **0.2.0** : Assign one split for each of the Kafka topic partitions. This makes Dataflow
37+
[Update](https://cloud.google.com/dataflow/pipelines/updating-a-pipeline)
38+
from previous version incompatible.
39+
* **0.1.0** : KafkaIO with support for Unbounded Source and Sink.

contrib/kafka/pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
<artifactId>google-cloud-dataflow-java-contrib-kafka</artifactId>
2626
<name>Google Cloud Dataflow Kafka IO Library</name>
2727
<description>Library to read Kafka topics.</description>
28-
<version>0.1.0-SNAPSHOT</version>
28+
<version>0.2.0-SNAPSHOT</version>
2929

3030
<properties>
3131
<dataflow.version>[1.6.0, 1.99)</dataflow.version>

0 commit comments

Comments
 (0)