High Metrics Cardinality Issue: Split `tidb_tikvclient_request_seconds` Metric to Reduce Time Series Explosion

[HaoW30]

Background

In large-scale deployments, the tidb_tikvclient_request_seconds histogram metric is a major contributor to metrics cardinality, accounting for approximately 25-30% of total time series. This creates significant operational overhead in metrics collection, storage, and querying.

Current Implementation

The metric is currently defined with four label dimensions:

TiKVSendReqHistogram = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Namespace:   namespace,
        Subsystem:   subsystem,
        Name:        "request_seconds",
        Help:        "Bucketed histogram of sending request duration.",
        Buckets:     prometheus.ExponentialBuckets(0.0005, 2, 24), // 0.5ms ~ 1.2h
        ConstLabels: constLabels,
    }, []string{LblType, LblStore, LblStaleRead, LblScope})

Cardinality Impact

This creates time series based on:

• ~40 request types (Cop, Get, Prewrite, Commit, etc.)
• N stores (number of TiKV instances)
• 2 stale_read values (true/false)
• 2 scope values (local/global)
• 24 histogram buckets

Total: ~3,840N time series (e.g., 384,000 time series for a 100-store cluster)

Root Cause

The metric combines orthogonal dimensions that are rarely queried together. In practice:

1. Cluster-wide analysis focuses on request types: "What's the P99 for Prewrite requests?"
2. Store-level analysis focuses on individual stores: "Is store X slow?"
3. Cross-dimensional queries (type X on store Y) are rare and better served by TiKV server-side metrics

The current design optimizes for the least common use case, causing massive cardinality explosion.

Proposed Solution

Split into two separate metrics:

1. By Request Type (cluster-wide)

// tidb_tikvclient_request_seconds
prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Namespace:   namespace,
        Subsystem:   subsystem,
        Name:        "request_seconds",
        Help:        "Bucketed histogram of sending request duration (cluster-wide).",
        Buckets:     prometheus.ExponentialBuckets(0.0005, 2, 24),
        ConstLabels: constLabels,
    }, []string{LblType, LblStaleRead, LblScope})

Cardinality: ~3,840 time series (constant, independent of cluster size)

2. By Store (aggregated across types)

// tidb_tikvclient_request_seconds_by_store
prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Namespace:   namespace,
        Subsystem:   subsystem,
        Name:        "request_seconds_by_store",
        Help:        "Bucketed histogram of sending request duration by store (aggregate across types).",
        Buckets:     prometheus.ExponentialBuckets(0.0005, 2, 24),
        ConstLabels: constLabels,
    }, []string{LblStore, LblStaleRead, LblScope})

Cardinality: ~96N time series

Impact

Before: ~3,840N time series
After: ~3,840 + 96N time series

For a 100-store cluster:

• Before: 384,000 time series
• After: 13,440 time series
• Reduction: ~96.5% (370,560 fewer time series)

Observability Impact

This change preserves observability for all common query patterns:

Use Case	Metric	Status
P99 by request type (cluster-wide)	tidb_tikvclient_request_seconds{type="Prewrite"}	✅ Unchanged
P99 by store (all types)	tidb_tikvclient_request_seconds_by_store{store="tikv-1"}	✅ New metric
P99 for type X on store Y	tikv_grpc_msg_duration_seconds (server-side)	✅ Use TiKV metric
Request rate by type	rate(tidb_tikvclient_request_seconds_count[5m])	✅ Unchanged
Store-level request rate	rate(tidb_tikvclient_request_seconds_by_store_count[5m])	✅ New metric

Implementation Approach

1. Add new metric tidb_tikvclient_request_seconds_by_store with {store, stale_read, scope} labels
2. Modify existing metric to remove LblStore, keeping {type, stale_read, scope} labels
3. Update all call sites that populate TiKVSendReqHistogram to record to both metrics appropriately
4. Update documentation and example queries

Why This Matters

This particularly benefits large TiDB clusters (50+ TiKV stores) where metrics cardinality becomes an operational challenge, causing:

• High OpenTelemetry collector CPU/memory usage
• Prometheus storage pressure and query performance issues
• Hitting cardinality limits in metrics backends

A single metric change can reduce total cluster time series by 25-30% in large deployments.

Alternatives Considered

1. Reduce histogram buckets - Rejected: Loses percentile accuracy, minimal reduction
2. Prometheus relabeling - Rejected: Pushes complexity to users, doesn't solve source overhead
3. Metric sampling - Rejected: Loses accuracy for critical performance metrics
4. Optional store label - Rejected: Doesn't address combinatorial explosion

---

Happy to contribute the implementation if this proposal is accepted.

[cfzjywxk]

The difference between the kv req duration at kv-client and the TiKV gRPC duration includes:

• Processing time at the kv-client and TiKV gRPC layers
• Latency in the network link layer
• Processing time at the batch-client layer implemented within kv-client

P99 for type X on store Y tikv_grpc_msg_duration_seconds (server-side) ✅ Use TiKV metric
If kv-client kv req duration is not recorded, it is theoretically impossible to understand the potential latency contributions of the above layers for a specific type of kv request.

A trade-off solution is to avoid aggregating all request types and instead aggregate based on commonly monitored categories, for example:

• Write type: Includes Prewrite | Commit | req — these requests require quorum writes on TiKV.
• Point Read: Includes PointGet req — corresponding to the shortest-path TiKV read.
• Cop Read: Includes Coprocessor req — corresponding to TiKV coprocessor read.
• Txn write: Includes HeartBeat | ResolveLock | PessimisticLock | CheckTxnStatus | CleanUp | PessimisticRollback — corresponding to transaction conflict handling requests on TiKV.
• Others

However, the trade-off is that this approach increases the difficulty of troubleshooting latency-related issues, because the essence of diagnosing latency problems lies in breaking down and accurately measuring the time spent at each stage.

[HaoW30]

@cfzjywxk I understand your point about keeping per-request-type metrics to track client-side latency, rather than server-side latency, in order to better understand and investigate the latency contributions of the layers you mentioned. That’s why I’m proposing to keep the following metric:

P99 by request type (cluster-wide) | tidb_tikvclient_request_seconds{type="Prewrite"} | ✅ Unchanged

My question is how often we actually need this metric at the per-store level. My understanding is that we want to track the processing time of these request types primarily at the cluster level, rather than per store.

For network latency issues, all request types should be affected, and those should be visible via:

P99 by store (all types) | tidb_tikvclient_request_seconds_by_store{store="tikv-1"} | ✅ New metric

Let me know if I’m missing a use case where per-request-type, per-store granularity would be necessary.

[cfzjywxk]

@HaoW30

Per-store level metrics are useful. For example, the execution latency of a commit statement typically depends on the slowest prewrite request, which is similarly recorded in slow logs.

For network latency issues, there are already relevant metrics in the TiDB panel, such as RPC layer latency.

I feel this is a difficult trade-off to balance: either sacrifice troubleshooting capability by reducing metrics and requirements, or break down and store more metrics to improve investigation efficiency.

Alternatively, if the abundance of metrics is indeed a significant issue, consider adding a switch or configuration. When enabled, this setting would minimize metric storage as much as possible.

[HaoW30]

Thanks @cfzjywxk! I think that’s a reasonable alternative. I can add a dynamically configurable switch for this. When enabled, we generate per-type, per-store metrics; when disabled, we fall back to split per-type + per-store metrics(as proposed here) to reduce cardinality. The option can be off by default and turned on only when deeper debugging is needed. Like, when identifying which store has the slowest prewrite requests.
Does this sound like a good plan?

[cfzjywxk]

@HaoW30

I think this configuration could be named something like kv-metric-strategy, offering the following options:

• Default, or per-type-per-store: This is the current mode with higher data volume and should be the default option.
• Split-per-store-per-type: This is the mode mentioned in the proposal to reduce the number of metrics.

Setting the default value to per-type-per-store and allowing dynamic, online modification is preferable to ensure:

1. Sufficient information is available for troubleshooting historical issues. Changing the value to true after a problem occurs would not capture prior historical data.
2. Compatibility with other diagnostic tools and code.

In clusters where metric storage load is high, the configuration can be adjusted to Split-per-store-per-type and changed back to Default or per-type-per-store as needed.

[HaoW30]

Ok, sounds good, will keep per-type-per-store metrics on by default. Thanks again @cfzjywxk !