Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pulsar-io] Add a Pulsar IO connector for InfluxDB sink #4017

Merged
merged 2 commits into from
Apr 11, 2019

Conversation

murong00
Copy link
Contributor

Motivation

Provides a builtin InfluxDB sink Connector, in order to persist pulsar messages to a InfluxDB database.

Modifications

Add a InfluxDB Sink and some unit tests.

Verifying this change

This change can be verified as follows:

  • deploy the InfluxDB Sink connector with configuration file containing the following fields:
configs:
    influxdbUrl: "http://localhost:8086"
    database: "test_db"
    consistencyLevel: "ONE"
    logLevel: "NONE"
    retentionPolicy: "autogen"
    gzipEnable: "false"
    batchTimeMs: "1000"
    batchSize: "100"
  • deploy an InfluxDB instance and create the above elements
  • send messages in the topic with specified schema declared when deploying the connector
  • use influx CLI to query messages in the specified database

@merlimat merlimat added this to the 2.4.0 milestone Apr 10, 2019

@FieldDoc(
required = false,
defaultValue = "",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should have sensitive=true

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


@FieldDoc(
required = false,
defaultValue = "",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should have sensitive=true

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

// create the database if not exists
List<String> databases = influxDB.describeDatabases();
if (!databases.contains(influxDatabase)) {
influxDB.createDatabase(influxDatabase);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a little suggestion:

if (!databases.contains(influxDatabase)) {
    if(influxDBSinkConfig.isCreateDatabase()) {
         influxDB.createDatabase(influxDatabase);
      }
      else {
        throw new IllegalArgumentException("This " + influxDBSinkConfig.getDatabase() + " database does not exist!");
      }
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ambition119 Thanks for the reviewing, it's not necessary to do this as field database is required and has been checked in method validate().

@murong00
Copy link
Contributor Author

run java8 tests

Copy link
Contributor

@merlimat merlimat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@merlimat merlimat merged commit 1d142cb into apache:master Apr 11, 2019
@murong00 murong00 deleted the branch-4017 branch March 8, 2020 04:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants