feat: Add ClickHouse driver to sql inputs/outputs plugins

[nahsi]

Required for all PRs:

• Updated associated README.md.
• Pull request title or commits are in conventional commit format (e.g. feat: or fix:)

Added ClickHouse to sql input drivers (using clickhouse-go)

---

Since there are no changes to plugin code, I'm not sure there is a need to add tests.

[telegraf-tiger]

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

[nahsi]

!signed-cla

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• darwin_amd64.tar.gz

• windows_amd64.zip

• windows_i386.zip

• linux_mips.tar.gz

• mips.deb

• static_linux_amd64.tar.gz

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

• linux_mipsel.tar.gz

• mipsel.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

[srebhan]

Looks good to me! Thanks for adding this driver @nahsi!

[srebhan]

@nahsi one additional request, can you please also prepare a pull-request for output/sql so we can keep the two symmetrical...

[srebhan]

@nahsi and another request: Please add a test-case, i.e. a docker instance where you can import the example tables. Take a look at the postgres and mysql tests in the repo.

[srebhan]

@nahsi any chance you add a test-case for this database type?

[nahsi]

@srebhan I will try to do it this weekend, very sorry for the delay 🙇🏼

[nahsi]

@srebhan to use clickhouse-go I had to use begin->prepare->exec->commit instead of just exec in outputs/sql.

With just exec clickhouse-go will emit this error.

I'm not sure if it is OK since I don't know golang and I know clickhouse only from OPS point of view :) But it works.

Maybe it would be better to begin then loop over metrics preparing sql etc and then commit? I don't know. If it is so I could try to update.

[srebhan]

Excellent work @nahsi, nice code and test! I have one suggestion regarding the special insert of clickhouse. My intention there is to prepare for other drivers that require special treatment by directly switch to a switch statement. What do you think?

[telegraf-tiger]

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_armv7.tar.gz
armhf.deb	i386.rpm	freebsd_i386.tar.gz
i386.deb	ppc64le.rpm	linux_amd64.tar.gz
mips.deb	s390x.rpm	linux_arm64.tar.gz
mipsel.deb	x86_64.rpm	linux_armel.tar.gz
ppc64el.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

@nahsi any update on this PR?

[nahsi]

Sorry, I will do it this week.

[nahsi]

@srebhan I've commited your suggestions, ready for another review.

[srebhan]

@nahsi thanks for the update. There are some minor formatting issues, but nothing too big.

[telegraf-tiger]

🥳 This pull request decreases the Telegraf binary size by -1.74 % for linux amd64 (new size: 134.1 MB, nightly size 136.5 MB)

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here ! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[nahsi]

@srebhan I've commited the changes you suggested. Please have a look again when you have time

[srebhan]

@nahsi perfect thank you for your contribution!

Looks good to me.

[crabvk]

Maybe I'm doing something wrong, but I'm getting this error

2022-02-14T05:32:39Z E! [agent] Failed to connect to [outputs.sql], retrying in 15s, error was 'code: 516, message: default: Authentication failed: password is incorrect or there is no user with such name'

Using following telegraf config

[[outputs.sql]]
  driver = "clickhouse"
  data_source_name = "clickhouse://user:password@example.com:9000/database"
  [outputs.sql.convert]
    integer = "Int64"
    text = "String"
    timestamp = "DateTime"
    defaultvalue = "String"
    unsigned = "UInt64"
    bool = "Uint8"

However, from the same server I'm able to connect to clickhouse using cli

clickhouse-client --user user --password password --host example.com --database database

I've checked credentials several times, they are the same in both cases.

UPD
Oops, I was using wrong connection string from clickhouse-go v2, for v1 it has different format

data_source_name = "tcp://example.com:9000?username=user&password=password&database=database"

Now connection works.
Thank you for the plugin 🙂

[crabvk]

Got another error now

2022-02-14T06:01:11Z E! [agent] Error writing to outputs.sql: code: 62, message: Syntax error: failed at position 72 ('UInt64'): UInt64,"inactive" Int64 UInt64,"vmalloc_chunk" Int64 UInt64,"sreclaimable" Int64 UInt64,"swap_cached" Int64 UInt64,"used" Int64 UInt64,"huge_pages_free" Int64 U. Expected one of: DEFAULT, MATERIALIZED, ALIAS, NOT, NULL, COMMENT, CODEC, TTL, token, Comma, ClosingRoundBracket

Clickhouse log message

2022.02.14 06:07:46.082356 [ 2196388 ] {1ff9aa5d-c0c5-4322-8b15-5e94806715ef} TCPHandler: Code: 62. DB::Exception: Syntax error: failed at position 72 ('UInt64'): UInt64,"inactive" Int64 UInt64,"vmalloc_chunk" Int64 UInt64,"sreclaimable" Int64 UInt64,"swap_cached" Int64 UInt64,"used" Int64 UInt64,"huge_pages_free" Int64 U. Expected one of: DEFAULT, MATERIALIZED, ALIAS, NOT, NULL, COMMENT, CODEC, TTL, token, Comma, ClosingRoundBracket. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0xa82d07a in /usr/bin/clickhouse
1. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, bool, unsigned long, unsigned long) @ 0x14d642bf in /usr/bin/clickhouse
2. ? @ 0x13d121f0 in /usr/bin/clickhouse
3. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum) @ 0x13d11e55 in /usr/bin/clickhouse
4. DB::TCPHandler::runImpl() @ 0x147f050f in /usr/bin/clickhouse
5. DB::TCPHandler::run() @ 0x14804259 in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x1745e52f in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x17460981 in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x17611609 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x1760ed00 in /usr/bin/clickhouse
10. ? @ 0x7f5257a61259 in ?
11. clone @ 0x7f525798a5e3 in ?

ClickHouse version 22.1.3.7

[crabvk]

I think the cause of the problem is here, when value type is uint64 clickhouse type becomes Int64 UInt64. The workaround is

  [outputs.sql.convert]
    integer = "BIGINT"
    unsigned = "UNSIGNED"

but it would be better setting

  [outputs.sql.convert]
    integer = "INT"
    unsigned = "INT UNSIGNED"

in the first place, because now unsigned in fact behaves more like unsigned_suffix.

Also, I've noticed that uptime column is not created in system table for the system input plugin.

[srebhan]

@crabvk can you please open an Issue for your issue ;-) instead of putting the information next to this PR!?

[Okaho]

its worked for me:

[[outputs.sql]]
   driver = "clickhouse"
   data_source_name = "tcp://example.com:9000?username=user&password=password&database=database"
   timestamp_column = "timestamp"
   table_template = "CREATE TABLE IF NOT EXISTS {TABLE}({COLUMNS}) ENGINE = MergeTree() PRIMARY KEY (timestamp,host) ORDER by (timestamp,host) Partition by toYYYYMMDD(timestamp)"
   table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"
  [outputs.sql.convert]
     integer              = "INT"
     real                 = "DOUBLE"
     text                 = "TEXT"
     timestamp            = "TIMESTAMP"
     defaultvalue         = "TEXT"
     unsigned             = "UNSIGNED"
     bool                 = "BOOL"

[Okaho]

Hi. What exactly interests you in telegrapr configuration? Something is wrong with CH? вт, 21 февр. 2023 г., 10:31 zentkhv ***@***.***>:

…

Can you please show you full telegraf config from you project? — Reply to this email directly, view it on GitHub <#9671 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALVWTURPGFZEAX3ZYF7AVZTWYQZINANCNFSM5CWWKZJA> . You are receiving this because you commented.Message ID: ***@***.***>

[zentkhv]

    HI! Im trying to send metrics to clickhouse by telegraf with you part config example, but geting an error: [agent] Failed to connect to [outputs.sql], retrying in 15s, error was 'driver: bad connection' Add my full config. I thought your full config will help me to solve my problem.      

Вторник, 21 февраля 2023, 15:00 +10:00 от Okaho ***@***.***>:     Hi. What exactly interests you in telegrapr configuration? Something is wrong with CH? вт, 21 февр. 2023 г., 10:31 zentkhv ***@***.***>: > Can you please show you full telegraf config from you project? > > — > Reply to this email directly, view it on GitHub > < #9671 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ALVWTURPGFZEAX3ZYF7AVZTWYQZINANCNFSM5CWWKZJA > > . > You are receiving this because you commented.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you commented. Message ID: <influxdata/telegraf/pull/9671/c1437859685 @ github . com>

 

[Okaho]

telegraf --version
Telegraf 1.24.0 (git: HEAD@3c4a6516)

/etc/telegraf/telegraf.conf

[agent]
  interval = "60s"
  metric_batch_size = 100
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "1s"
  flush_jitter = "0s"
  precision = "0s"
  logfile = "/var/log/telegraf/telegraf.log"

  hostname = "host"
  omit_hostname = false

[[outputs.sql]]
   driver = "clickhouse"

data_source_name = "tcp://aa.bb.cc.dd:9000?username=username&password=password&database=telegraf3"

   timestamp_column = "timestamp"
   table_template = "CREATE TABLE IF NOT EXISTS {TABLE}({COLUMNS}) ENGINE = MergeTree() PRIMARY KEY (timestamp,host) ORDER by (timestamp,host) Partition by toYYYYMMDD(timestamp)"
    table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"
  [outputs.sql.convert]
     integer              = "INT"
     real                 = "DOUBLE"
     text                 = "TEXT"
     timestamp            = "TIMESTAMP"
     defaultvalue         = "TEXT"
     unsigned             = "UNSIGNED"
     bool                 = "BOOL"


[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

[[inputs.disk]]
   mount_points = ["/"]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

/etc/telegraf/telegraf.d/ping-telegraf.conf

[[inputs.ping]]
    count = 30
    ping_interval = 2.0
    timeout = 0.9
    deadline = 58
    urls = [
    "localhost",
    "gateway"
]

ClickHouse:

SHOW CREATE TABLE telegraf3.ping
 CREATE TABLE telegraf3.ping
(
    `timestamp` DateTime,
    `host` String,
    `url` String,
    `standard_deviation_ms` Float64,
    `packets_transmitted` Int32,
    `packets_received` Int32,
    `percent_packet_loss` Float64,
    `ttl` Int32,
    `minimum_response_ms` Float64,
    `maximum_response_ms` Float64,
    `result_code` Int32,
    `average_response_ms` Float64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
PRIMARY KEY (timestamp, host)
ORDER BY (timestamp, host)
SETTINGS index_granularity = 8192

--

CREATE TABLE telegraf3.disk
(
    `timestamp` DateTime,
    `device` String,
    `fstype` String,
    `host` String,
    `mode` String,
    `path` String,
    `used_percent` Float64,
    `inodes_total` UInt32,
    `inodes_free` UInt32,
    `inodes_used` UInt32,
    `total` UInt32,
    `free` UInt32,
    `used` UInt32
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
PRIMARY KEY (timestamp, host)
ORDER BY (timestamp, host)
SETTINGS index_granularity = 8192

--

CREATE TABLE telegraf3.cpu
(
    `timestamp` DateTime,
    `cpu` String,
    `host` String,
    `usage_system` Float64,
    `usage_iowait` Float64,
    `usage_irq` Float64,
    `usage_steal` Float64,
    `usage_guest_nice` Float64,
    `usage_user` Float64,
    `usage_idle` Float64,
    `usage_nice` Float64,
    `usage_softirq` Float64,
    `usage_guest` Float64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
PRIMARY KEY (timestamp, host)
ORDER BY (timestamp, host)
SETTINGS index_granularity = 8192

SELECT *
FROM telegraf3.ping
WHERE (url = 'localhost') 
LIMIT 1

┌───────────timestamp─┬─host─┬─url───────┬─standard_deviation_ms─┬─packets_transmitted─┬─packets_received─┬─percent_packet_loss─┬─ttl─┬─minimum_response_ms─┬─maximum_response_ms─┬─result_code─┬─average_response_ms─┐
│ 2022-12-11 00:00:50 │ host │ localhost │                 0.155 │                  30 │               30 │                   0 │  64 │               0.021 │               1.435 │           1 │               0.871 │
└─────────────────────┴──────┴───────────┴───────────────────────┴─────────────────────┴──────────────────┴─────────────────────┴─────┴─────────────────────┴─────────────────────┴─────────────┴─────────────────────┘

[Okaho]

@zentkhv, did you succeed?

[zentkhv]

@Okaho , How did you understand which fields need to be added to the clickhouse table structure?
p.s. it seems from the output:
cpu,cpu=cpu0,host=server usage_idle=85.86497890295448,usage_nice=0,usage_iowait=0.10548523206751279,usage_irq=0,usage_guest_nice=0,usage_user=7.8059071729957115,usage_system=6.223628691983146,usage_guest=0,usage_softirq=0,usage_steal=0 1678926350000000000

[Okaho]

My tables were created by telegraf, the above structures are given as an example. I suggest deleting the tables. The telegraf version is very important, i have 1.24.0

[zentkhv]

@Okaho , thank you very much! Finally I managed to write the data. I have a telegraph version 1.25.0, it writes both to a pre-created table, and can create its own.

[Okaho]

Great, very glad it worked out for you!

[Okaho]

Help me set up a route to send data to different systems. Using "routing plugin" https://www.influxdata.com/blog/telegraf-best-practices/ data stops going to both output plugin.

[majlsfiles]

@Okaho , thank you very much! Finally I managed to write the data. I have a telegraph version 1.25.0, it writes both to a pre-created table, and can create its own.

I am using telegraf 1.29.4 and ClickHouse 24.1.5.6, while everything works great I have problem that metrics are written slow and after a while buffer is full and metrics are dropped.

2024-02-21T09:31:04+01:00 D! [outputs.sql] Wrote batch of 1000 metrics in 6.234639174s

This should probably be in ms and not in seconds. Can someone please help with this issue?

[powersj]

Can someone please help with this issue?

Please use the slack of forums to get support and not comment on closed PRs. If sending 1000 metrics is slow, then consider getting a network trace to see what part of the request is taking a long time, as most of the time the output spends is using the network.