-
Notifications
You must be signed in to change notification settings - Fork 8
Kafka Connect GCP Bigtable sink configurations
Sanju Thomas edited this page Oct 6, 2019
·
1 revision
Property | Value | Data Type | Description |
---|---|---|---|
name | bigtable-sink | String | Name of the Sink Connect. |
connector.class | BigtableSinkConnector | String | Simple name of the Connector Class. |
tasks.max | 1 | Number | Numbers of tasks. |
topics | demo-topic | String | Comma separated list of topics. |
config.files.location | kafka_home/config | String | Drop the topic to table mapping in this folder. File name must be topic-name.yml |
Property | Value | Data Type | Description |
---|---|---|---|
keyFile | /home/keys/demo-instance-key.json | String | GCP Connect Key File. This is a topic level configuration because you could subscribe from multiple topics and messages from one topic may go to a table in instance A and messages from another topic may go to a table in instance B |
project | demo-project | String | Name of the GCP Project |
instance | demo-instance | String | Name of GCP Bigtable instance |
table | demo-table | String | Name of GCP Bigtable table |
bulkMutateRowsMaxSize | 1024 (Optional, default is 1024) | Number | Maximum number of rows in a bulk mutation request |
continueAfterWriteError | false (Optional, default is false) | Boolean | Continue the write after a batch is error-ed out. |
transformer | com.sanjuthomas.gcp.transform.JsonEventTransformer | String | Transformer class to transform the message to Bigtable writable row. You may provide your own implementation. |
keyQualifiers | - exchange - symbol |
Array (Optional, default is message key.) | Bigtable row key qualifier. Configured element names would be used to construct the row keys. |
keyDelimiter | - | String (Optional, default is empty.) | Delimiter to use if there are more than one element to construct row key. |
families | - data - metadata |
Array | Column families in the Bigtable table. This configuration will be used by the transformer. |
familyQualifiers | - data: - client - exchange - symbol - price - quantity - metadata: - created_at - processed_at - topic - partition |
Array | Column family to columns mapping. |
- side bar