Update ClickHouse mixin to better utilize libraries #1445

schmikei · 2025-05-27T17:39:41Z

Utilizing common-lib, logs-lib and grafonnet for most of the mixin generation
Following inspiration for more recently updated mixins like Vsphere

We need to retake some screenshots I believe for this update as now logs are utilizing the logs-lib and will be on a separate dashboard now instead of baked into the overview.

Clickhouse Overview

Clickhouse Latency

Clickhouse replica

Logs

v-zhuravlev

Thanks for contributing! Small comment below while still a draft.

v-zhuravlev · 2025-05-29T12:42:29Z

clickhouse-mixin/dashboards.libsonnet

+              panels.interserverConnectionsPanel { gridPos+: { h: 8, w: 12, x: 0, y: 0 } },
+              panels.replicaQueueSizePanel { gridPos+: { h: 8, w: 12, x: 12, y: 0 } },
+              panels.replicaOperationsPanel { gridPos+: { h: 8, w: 12, x: 0, y: 8 } },
+              panels.replicaReadOnlyPanel { gridPos+: { h: 8, w: 12, x: 12, y: 8 } },
+              panels.zooKeeperWatchesPanel { gridPos+: { h: 8, w: 12, x: 0, y: 16 } },
+              panels.zooKeeperSessionsPanel { gridPos+: { h: 8, w: 12, x: 12, y: 16 } },
+              panels.zooKeeperRequestsPanel { gridPos+: { h: 8, w: 24, x: 0, y: 24 } },


x,y coordinates pair is redundant if wrapPanels is used. It will be calculated.

Sweet thank you for the tip! 🚀

…t-libs into clickhouse-modernization

v-zhuravlev

Thanks! Some comments inline.

v-zhuravlev · 2025-06-18T17:54:33Z

clickhouse-mixin/rows.libsonnet

+  new(panels): {
+    row_1:


seems like rows are not used to build dashboards.

v-zhuravlev · 2025-06-18T17:55:20Z

clickhouse-mixin/targets.libsonnet

+        'rate(ClickHouseProfileEvents_RejectedInserts{%s}[$__rate_interval])' % selector
+      )
+      + g.panel.timeSeries.queryOptions.withInterval('30s')
+      + prometheusQuery.withLegendFormat('%s - Rejected Inserts' % utils.labelsToPanelLegend(config.legendLabels)),


Suggested change

+ prometheusQuery.withLegendFormat('%s - Rejected Inserts' % utils.labelsToPanelLegend(config.legendLabels)),

+ prometheusQuery.withLegendFormat('%s - Rejected Inserts' % utils.labelsToPanelLegend(config.legendLabels)),

here and multiple places.
Please use Sentence case.

v-zhuravlev · 2025-06-18T17:56:38Z

clickhouse-mixin/targets.libsonnet

+    ClickHouseProfileEvents_DiskReadElapsedMicroseconds:
+      prometheusQuery.new(
+        '${' + vars.datasources.prometheus.name + '}',
+        'increase(ClickHouseProfileEvents_DiskReadElapsedMicroseconds{%s}[$__rate_interval])' % selector


Suggested change

'increase(ClickHouseProfileEvents_DiskReadElapsedMicroseconds{%s}[$__rate_interval])' % selector

'increase(ClickHouseProfileEvents_DiskReadElapsedMicroseconds{%s}[$__rate_interval])' % selector

to get accurate increments, Increase() should be used with [$__interval:] -offset $__interval.
rate() is best used with $__rate_interval, but it gives per second ratio.

v-zhuravlev · 2025-06-18T17:57:54Z

clickhouse-mixin/variables.libsonnet

this whole file could be ommited. Its contents could generated with commonlib.variables.new()

v-zhuravlev · 2025-06-18T17:58:40Z

clickhouse-mixin/jsonnetfile.json

-          "remote": "https://github.com/grafana/jsonnet-libs.git",
-          "subdir": "grafana-cloud-integration-utils"
+          "remote": "https://github.com/grafana/grafonnet.git",
+          "subdir": "gen/grafonnet-v10.0.0"


Suggested change

"subdir": "gen/grafonnet-v10.0.0"

"subdir": "gen/grafonnet-v11.0.0"

best to use v11.

v-zhuravlev · 2025-06-18T17:59:50Z

clickhouse-mixin/dashboards.libsonnet

+    local panels = this.grafana.panels;
+
+    {
+      'clickhouse-replica':


Suggested change

'clickhouse-replica':

'clickhouse-replica.json':

.json suffix should be used for dashboards.

v-zhuravlev · 2025-06-18T18:01:44Z

clickhouse-mixin/config.libsonnet

-  _config+:: {
-    enableMultiCluster: false,
-    clickhouseSelector: if self.enableMultiCluster then 'job=~"$job", instance=~"$instance", cluster=~"$cluster"' else 'job=~"$job", instance=~"$instance"',
+  enableMultiCluster: false,


Suggested change

enableMultiCluster: false,

enableMultiCluster: false,

overall, I think cluster could be considered a specific case of a member of groupLabel. Do we need any specific support for clusterLabel?

v-zhuravlev · 2025-06-18T18:03:20Z

clickhouse-mixin/targets.libsonnet

+local utils = commonlib.utils {
+  labelsToPanelLegend(labels): std.join(' - ', ['{{%s}}' % [label] for label in labels]),
+};


I think this function is present in commonlib.utils now?

v-zhuravlev · 2025-06-18T18:03:41Z

clickhouse-mixin/targets.libsonnet

Is it possible to transform targets to signals?

update clickhouse to latest mixin standards

f8e0483

v-zhuravlev reviewed May 29, 2025

View reviewed changes

schmikei and others added 6 commits May 29, 2025 13:49

remove redundant wrapPanels

0261e61

Merge branch 'master' into clickhouse-modernization

b92345f

cleanup

1110855

Merge branch 'master' into clickhouse-modernization

04709db

fix variable setting

349279a

Merge branch 'clickhouse-modernization' of github.com:schmikei/jsonne…

0993930

…t-libs into clickhouse-modernization

schmikei marked this pull request as ready for review June 10, 2025 20:21

schmikei requested a review from a team as a code owner June 10, 2025 20:21

schmikei changed the title ~~WIP: Update ClickHouse mixin to better utilize libraries~~ Update ClickHouse mixin to better utilize libraries Jun 10, 2025

remove unused variables

25f9197

v-zhuravlev added the monitoring-mixins label Jun 18, 2025

v-zhuravlev requested changes Jun 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update ClickHouse mixin to better utilize libraries #1445

Update ClickHouse mixin to better utilize libraries #1445

Uh oh!

schmikei commented May 27, 2025 •

edited

Loading

Uh oh!

v-zhuravlev left a comment •

edited

Loading

Uh oh!

v-zhuravlev May 29, 2025

Uh oh!

schmikei May 29, 2025

Uh oh!

v-zhuravlev left a comment

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

v-zhuravlev Jun 18, 2025

Uh oh!

Uh oh!

	+ prometheusQuery.withLegendFormat('%s - Rejected Inserts' % utils.labelsToPanelLegend(config.legendLabels)),
	+ prometheusQuery.withLegendFormat('%s - Rejected Inserts' % utils.labelsToPanelLegend(config.legendLabels)),

	'increase(ClickHouseProfileEvents_DiskReadElapsedMicroseconds{%s}[$__rate_interval])' % selector
	'increase(ClickHouseProfileEvents_DiskReadElapsedMicroseconds{%s}[$__rate_interval])' % selector

	"subdir": "gen/grafonnet-v10.0.0"
	"subdir": "gen/grafonnet-v11.0.0"

Update ClickHouse mixin to better utilize libraries #1445

Are you sure you want to change the base?

Update ClickHouse mixin to better utilize libraries #1445

Uh oh!

Conversation

schmikei commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

v-zhuravlev left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v-zhuravlev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

schmikei commented May 27, 2025 •

edited

Loading

v-zhuravlev left a comment •

edited

Loading