Lack of Configuration for Metrics Prefix with Distributed Tables in Grafana ClickHouse Plugin

[mlsalcedo]

Description

We've encountered a limitation with the Grafana ClickHouse plugin where there's no option to configure a metrics prefix for use with ClickHouse's distributed tables. This limitation is leading to a significant issue where some shards are running out of space due to uneven distribution of metrics data.

Expected Behavior

Ideally, the plugin should provide a configuration option to specify a metrics prefix. This would allow users to efficiently distribute their data across multiple shards in a ClickHouse cluster, preventing individual shards from running out of space and ensuring balanced data distribution.

Actual Behavior

Currently, there is no option to set a metrics prefix for distributed tables within the Grafana ClickHouse plugin. As a result, data is not evenly distributed across the shards, causing some shards to quickly run out of space while others remain underutilized.

Steps to Reproduce

1. Set up Grafana with the ClickHouse plugin and configure it to connect to a ClickHouse cluster.
2. Insert metrics data without the ability to specify a prefix for distributed tables.
3. Observe the uneven data distribution across the shards, leading to space issues in certain shards.

Possible Solution

Implementing a feature within the plugin's configuration settings to specify a metrics prefix for distributed tables would likely resolve this issue. This addition would provide greater control over data distribution in a ClickHouse cluster and help prevent uneven shard usage.

Environment

• Grafana version: 10.3.3
• ClickHouse plugin version: 24.x

[SpencerTorres]

I see your ClickHouse version is 24.x, but what is your plugin version? Latest is at 4.0.4 as of right now

[mlsalcedo]

A similar issues was addressed in 704 and plugin version is 4.0.4

[SpencerTorres]

Could you share some example queries? Such as what the current query is vs what the expected query is.

Also, are you using the query builder or SQL editor?

[mlsalcedo]

The issue is mostly with the inserts, there's not place in the configuration to set the prefix for the metric's tables.

[SpencerTorres]

Apologies for not understanding the issue, I appreciate your patience.

What do you mean by inserts? Are you inserting data through Grafana?

Is this referring to the open telemetry exporter by chance? In that case you can configure the prefix for otel metrics, it defaults to otel_metrics. The Grafana plugin has no special support for OTEL metrics, those tables are treated as any other table in ClickHouse.

If this is an issue or feature request for the open telemetry exporter, then it is better to create it on that repo. The Grafana plugin simply queries the data inserted from there.

Other links:

• Create new issue for OTEL exporter (choose exporter/clickhouse)
• OTEL Exporter Metrics table suffixes
• OTEL Exporter configuration options (see metrics_table_name)
• OTEL exporter recently added support for ON CLUSTER

[mlsalcedo]

You are correct, I will close the ticket. Do we have any plans of supporting ServiceGraph?

[mlsalcedo]

Closing since it's not issue