Skip to content

Commit 2c679c3

Browse files
committed
Merge remote-tracking branch 'upstream/master'
2 parents a56c28d + 1eb4c26 commit 2c679c3

File tree

46 files changed

+2505
-1029
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+2505
-1029
lines changed

docs/learn/documentation/versioned/azure/eventhubs.md

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
layout: page
3-
title: Connecting to Eventhubs
3+
title: Connecting to Event Hubs
44
---
55
<!--
66
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -19,13 +19,13 @@ title: Connecting to Eventhubs
1919
limitations under the License.
2020
-->
2121

22-
You can configure your Samza jobs to process data from [Azure Eventhubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features), Microsoft's data streaming service. An `event hub` is similar to a Kafka topic and can have multiple partitions with producers and consumers. Each message produced or consumed from an event hub is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data).
22+
You can configure your Samza jobs to process data from [Azure Event Hubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features), Microsoft's data streaming service. An `event hub` is similar to a Kafka topic and can have multiple partitions with producers and consumers. Each message produced or consumed from an event hub is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data).
2323

24-
### Consuming from EventHubs:
24+
### Consuming from Event Hubs:
2525

2626
Samza's [EventHubSystemConsumer](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubSystemConsumer.java) wraps the EventData into an [EventHubIncomingMessageEnvelope](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubIncomingMessageEnvelope.java). The key of the message is set to the partition key of the EventData. The message is obtained from the EventData body.
2727

28-
To configure Samza to configure from EventHub streams:
28+
To configure Samza to configure from Event Hubs streams:
2929

3030
```
3131
# define an event hub system factory with your identifier. eg: eh-system
@@ -46,21 +46,21 @@ systems.eh-system.streams.output0.eventhubs.sas.keyname=YOUR-SAS-KEY-NAME
4646
systems.eh-system.streams.output0.eventhubs.sas.token=YOUR-SAS-KEY-TOKEN
4747
```
4848

49-
The tuple required to access the Eventhubs entity per stream must be provided, namely the fields `YOUR-STREAM-NAMESPACE`, `YOUR-ENTITY-NAME`, `YOUR-SAS-KEY-NAME`, `YOUR-SAS-KEY-TOKEN`.
49+
The tuple required to access the Event Hubs entity per stream must be provided, namely the fields `YOUR-STREAM-NAMESPACE`, `YOUR-ENTITY-NAME`, `YOUR-SAS-KEY-NAME`, `YOUR-SAS-KEY-TOKEN`.
5050

51-
### Producing to EventHubs:
51+
### Producing to Event Hubs:
5252

53-
Similarly, you can also configure your Samza job to write to EventHubs.
53+
Similarly, you can also configure your Samza job to write to Event Hubs.
5454
```
5555
OutgoingMessageEnvelope envelope = new OutgoingMessageEnvelope(new SystemStream("eh-system", "output0"), key, message);
5656
collector.send(envelope);
5757
```
5858

59-
Each [OutgoingMessageEnvelope](https://samza.apache.org/learn/documentation/latest/api/javadocs/org/apache/samza/system/OutgoingMessageEnvelope.html) is converted into an [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data) instance whose body is set to the `message` in the envelope. Additionally, the `key` and the `produce timestamp` are set as properties in the EventData before sending it to EventHubs.
59+
Each [OutgoingMessageEnvelope](https://samza.apache.org/learn/documentation/latest/api/javadocs/org/apache/samza/system/OutgoingMessageEnvelope.html) is converted into an [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data) instance whose body is set to the `message` in the envelope. Additionally, the `key` and the `produce timestamp` are set as properties in the EventData before sending it to Event Hubs.
6060

6161
#### Size limit of partition key:
6262

63-
Note that EventHubs has a limit on the length of partition key (128 characters). In [EventHubSystemProducer](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/producer/EventHubSystemProducer.java) we truncate the partition key if the size of the key exceeds the limit.
63+
Note that Event Hubs has a limit on the length of partition key (128 characters). In [EventHubSystemProducer](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/producer/EventHubSystemProducer.java) we truncate the partition key if the size of the key exceeds the limit.
6464

6565
### Advanced configuration:
6666

@@ -82,15 +82,15 @@ systems.eh-system.partition.method = EVENT_HUB_HASHING
8282

8383
##### Consumer groups:
8484

85-
Eventhub supports a notion of [consumer groups](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups) which enable multiple applications have their own view of the event stream. Each event hub stream has a pre-defined consumer group named `$Default`. You can define your own consumer group for your job and configure a `eventhubs.consumer.group`
85+
Event Hubs supports a notion of [consumer groups](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups) which enable multiple applications have their own view of the event stream. Each event hub stream has a pre-defined consumer group named `$Default`. You can define your own consumer group for your job and configure a `eventhubs.consumer.group`
8686

8787
```
8888
systems.eh-system.streams.eh-input0.eventhubs.consumer.group = my-group
8989
```
9090

9191
##### Serde:
9292

93-
By default, the messages from EventHubs are sent and received as byte arrays. You can configure a serializer and deserializer for your message by setting a value for `msg.serde` for your stream.
93+
By default, the messages from Event Hubs are sent and received as byte arrays. You can configure a serializer and deserializer for your message by setting a value for `msg.serde` for your stream.
9494

9595
```
9696
streams.input0.samza.msg.serde = json
@@ -107,9 +107,9 @@ systems.eh-system.eventhubs.receive.queue.size = 10
107107

108108
For the list of all configs, check out the configuration table page [here](../jobs/configuration-table.html)
109109

110-
### Azure Eventhubs Hello-Samza Example
110+
### Azure Event Hubs Hello-Samza Example
111111

112-
The [hello-samza](https://github.com/apache/samza-hello-samza) project contains an example of a high level job that consumes and produces to Eventhub using the Zookeeper deployment model.
112+
The [hello-samza](https://github.com/apache/samza-hello-samza) project contains an example of a high level job that consumes and produces to Event Hubs using the Zookeeper deployment model.
113113

114114
#### Get the Code
115115

@@ -141,13 +141,13 @@ If you get a complaint that JAVA_HOME is not set, then you'll need to set it to
141141
Here are the configs you must set before building the project. Configure these in the `src/main/config/azure-application-local-runner.properties` file.
142142

143143
```
144-
# Add your EventHubs input stream credentials here
144+
# Add your Event Hubs input stream credentials here
145145
systems.eventhubs.streams.input-stream.eventhubs.namespace=YOUR-STREAM-NAMESPACE
146146
systems.eventhubs.streams.input-stream.eventhubs.entitypath=YOUR-ENTITY-NAME
147147
systems.eventhubs.streams.input-stream.eventhubs.sas.keyname=YOUR-SAS-KEY-NAME
148148
systems.eventhubs.streams.input-stream.eventhubs.sas.token=YOUR-SAS-KEY-TOKEN
149149
150-
# Add your EventHubs output stream credentials here
150+
# Add your Event Hubs output stream credentials here
151151
systems.eventhubs.streams.output-stream.eventhubs.namespace=YOUR-STREAM-NAMESPACE
152152
systems.eventhubs.streams.output-stream.eventhubs.entitypath=YOUR-ENTITY-NAME
153153
systems.eventhubs.streams.output-stream.eventhubs.sas.keyname=YOUR-SAS-KEY-NAME

docs/learn/documentation/versioned/connectors/eventhubs.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
layout: page
3-
title: Eventhubs Connector
3+
title: Event Hubs Connector
44
---
55
<!--
66
Licensed to the Apache Software Foundation (ASF) under one or more
@@ -21,13 +21,13 @@ title: Eventhubs Connector
2121

2222
## Overview
2323

24-
The Samza EventHubs connector provides access to [Azure Eventhubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features), Microsoft’s data streaming service on Azure. An event hub is similar to a Kafka topic and can have multiple partitions with producers and consumers. Each message produced or consumed from an event hub is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data).
24+
The Samza Event Hubs connector provides access to [Azure Event Hubs](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features), Microsoft’s data streaming service on Azure. An event hub is similar to a Kafka topic and can have multiple partitions with producers and consumers. Each message produced or consumed from an event hub is an instance of [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data).
2525

26-
## Consuming from EventHubs
26+
## Consuming from Event Hubs
2727

28-
Samza’s [EventHubSystemConsumer](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubSystemConsumer.java) wraps the EventData into an [EventHubIncomingMessageEnvelope](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubIncomingMessageEnvelope.java). Samza's eventhubs consumer wraps each message from Eventhubs into an EventHubMessageEnvelope. The envelope has two fields of interest - the key, which is set to the event's partition key and the message, which is set to the actual data in the event.
28+
Samza’s [EventHubSystemConsumer](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubSystemConsumer.java) wraps the EventData into an [EventHubIncomingMessageEnvelope](https://github.com/apache/samza/blob/master/samza-azure/src/main/java/org/apache/samza/system/eventhub/consumer/EventHubIncomingMessageEnvelope.java). Samza's Event Hubs consumer wraps each message from Event Hubs into an EventHubMessageEnvelope. The envelope has two fields of interest - the key, which is set to the event's partition key and the message, which is set to the actual data in the event.
2929

30-
You can configure your Samza jobs to process data from Azure Eventhubs. To configure Samza to consume from EventHub streams:
30+
You can configure your Samza jobs to process data from Azure Event Hubs. To configure Samza to consume from Event Hubs streams:
3131

3232
{% highlight jproperties %}
3333
# define an event hub system factory with your identifier. eg: eh-system
@@ -46,9 +46,9 @@ streams.eh-input-stream.eventhubs.sas.token=YOUR-SAS-KEY-TOKEN
4646

4747
It is required to provide values for YOUR-STREAM-NAMESPACE, YOUR-ENTITY-NAME, YOUR-SAS-KEY-NAME, YOUR-SAS-KEY-TOKEN to read or write to the stream.
4848

49-
## Producing to EventHubs
49+
## Producing to Event Hubs
5050

51-
Similarly, you can also configure your Samza job to write to EventHubs. Follow the same configs defined in the Consuming from EventHubs section to write to EventHubs:
51+
Similarly, you can also configure your Samza job to write to Event Hubs. Follow the same configs defined in the Consuming from Event Hubs section to write to Event Hubs:
5252

5353
{% highlight jproperties %}
5454
# define an event hub system factory with your identifier. eg: eh-system
@@ -64,14 +64,14 @@ streams.eh-output-stream.eventhubs.sas.keyname=YOUR-SAS-KEY-NAME
6464
streams.eh-output-stream.eventhubs.sas.token=YOUR-SAS-KEY-TOKEN
6565
{% endhighlight %}
6666

67-
Then you can create and produce a message to eventhubs in your code as below:
67+
Then you can create and produce a message to Event Hubs in your code as below:
6868

6969
{% highlight java %}
7070
OutgoingMessageEnvelope envelope = new OutgoingMessageEnvelope(new SystemStream("eh-system", "output0"), key, message);
7171
collector.send(envelope);
7272
{% endhighlight %}
7373

74-
Each [OutgoingMessageEnvelope](https://samza.apache.org/learn/documentation/latest/api/javadocs/org/apache/samza/system/OutgoingMessageEnvelope.html) is converted into an [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data) instance whose body is set to the message in the envelope. Additionally, the key and the produce timestamp are set as properties in the EventData before sending it to EventHubs.
74+
Each [OutgoingMessageEnvelope](https://samza.apache.org/learn/documentation/latest/api/javadocs/org/apache/samza/system/OutgoingMessageEnvelope.html) is converted into an [EventData](https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs._event_data) instance whose body is set to the message in the envelope. Additionally, the key and the produce timestamp are set as properties in the EventData before sending it to Event Hubs.
7575

7676
## Advanced configuration
7777

@@ -91,15 +91,15 @@ systems.eh-system.partition.method = EVENT_HUB_HASHING
9191

9292
### Consumer groups
9393

94-
Eventhub supports the notion of [consumer groups](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups) which enable multiple applications to have their own view of the event stream. Each partition is exclusively consumed by one consumer in the consumer group. Each event hub stream has a pre-defined consumer group named $Default. You can define your own consumer group for your job by configuring a eventhubs.consumer.group
94+
Event Hubs supports the notion of [consumer groups](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups) which enable multiple applications to have their own view of the event stream. Each partition is exclusively consumed by one consumer in the consumer group. Each event hub stream has a pre-defined consumer group named $Default. You can define your own consumer group for your job by configuring an Event Hubs.consumer.group
9595

9696
{% highlight jproperties %}
9797
streams.eh-input-stream.eventhubs.consumer.group = my-group
9898
{% endhighlight %}
9999

100100
### Serde
101101

102-
By default, the messages from EventHubs are sent and received as byte arrays. You can configure a serializer and deserializer for your message by setting a value for msg.serde for your stream.
102+
By default, the messages from Event Hubs are sent and received as byte arrays. You can configure a serializer and deserializer for your message by setting a value for msg.serde for your stream.
103103

104104
{% highlight jproperties %}
105105
streams.input0.samza.msg.serde = json

0 commit comments

Comments
 (0)