Health reporting cache reports false leaderless/under-replicated partitions, causing sidecar readiness probe failures

[AldoFusterTurpin]

Version & Environment

Redpanda version: v26.1.4 (commit 076e88a8e87668341e966bdfe2755a9dcbcf4a7f)

• Kubernetes: server v1.32.10, client v1.31.0
• Deployed via Redpanda Operator on self-hosted Kubernetes on AWS (r5a.2xlarge EC2 instances, EBS-backed NVMe via Nitro)
• 3-broker StatefulSet, RF=3, SASL+TLS enabled
• OS: Linux 6.16.4-zabbly+ (Ubuntu 24.04, #ubuntu24.04 SMP PREEMPT_DYNAMIC Fri Aug 29 00:54:49 UTC 2025 x86_64)
• Storage: EBS gp3, XFS filesystem
• No Kafka client libraries — issue is observed via rpk and Admin API only

What went wrong?

rpk cluster health, /v1/cluster/health_overview, the Redpanda Console UI, and Prometheus metrics (redpanda_kafka_under_replicated_replicas) all consistently report leaderless and under-replicated partitions when the cluster is actually healthy.

This has happened repeatedly (at least 3 times in 2 days) and persists for extended periods (many hours) without resolving. All three brokers are up and running throughout.

Example output from rpk cluster health while the issue is occurring:

CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          false
Unhealthy reasons:                [leaderless_partitions under_replicated_partitions]
Controller ID:                    2
All nodes:                        [0 1 2]
Nodes down:                       []
Nodes in recovery mode:           []
Nodes with high disk usage:       []
Leaderless partitions (16):       [kafka/__consumer_offsets/0 kafka/__consumer_offsets/5 kafka/__consumer_offsets/8 kafka/_schemas/0 kafka/block.io.req.v1/0 kafka/fdb.trace-logs.v1/0 kafka/fdb.trace-logs.v1/2 kafka/fdb.trace-logs.v1/4 kafka/fdb.trace-logs.v1/6 kafka/fdb.trace-logs.v1/7 kafka/fdb.trace-logs.v1/8 kafka/k8s.events/0 kafka/mercury.v1/0 kafka/mercury.v4/0 kafka/owl.enriched.full/0 kafka/pg.log_metadata.v2/0]
Under-replicated partitions (8):  [kafka/__consumer_offsets/0 kafka/__consumer_offsets/10 kafka/__consumer_offsets/14 kafka/__consumer_offsets/2 kafka/__consumer_offsets/5 kafka/__consumer_offsets/7 kafka/__consumer_offsets/8 kafka/fdb.trace-logs.v1/4]

At the same time, /v1/cluster/health_overview also reports is_healthy: false with the same leaderless/under-replicated lists.

However, querying individual partitions via the Admin API shows all partitions have valid leaders:

$ kubectl exec -n savannah-system redpanda-0 -c redpanda -- \
    curl -sk "https://localhost:9644/v1/partitions/kafka/owl.enriched.full/0" | python3 -m json.tool
{
    "ns": "kafka",
    "topic": "owl.enriched.full",
    "partition_id": 0,
    "status": "done",
    "leader_id": 2,
    "raft_group_id": 58,
    "replicas": [
        {"node_id": 1, "core": 0},
        {"node_id": 2, "core": 0},
        {"node_id": 0, "core": 0}
    ],
    "disabled": false
}

We verified this for every partition reported as leaderless — all returned a valid leader_id and status: done.

As definitive proof, we produced directly to four of the "leaderless" topics while the health endpoints were still reporting them as leaderless. All produces succeeded immediately with no errors:

=== fdb.trace-logs.v1 ===
Produced to partition 2 at offset 0 with timestamp 1778079854916.
=== owl.enriched.full ===
Produced to partition 0 at offset 1 with timestamp 1778079857260.
=== block.io.req.v1 ===
Produced to partition 0 at offset 0 with timestamp 1778079865262.
=== pg.log_metadata.v2 ===
Produced to partition 0 at offset 0 with timestamp 1778079867639.

A genuinely leaderless partition would return LEADER_NOT_AVAILABLE immediately — these all succeeded, confirming the health reporting is stale.

The Redpanda Console UI shows the same stale data under Cluster Health Debug:

Reason:                  Leaderless partitions, under-replicated partitions
Unreachable brokers:     0
Leaderless partitions:   16  View topics
Under-replicated partitions: 11  View topics

This confirms that the Console is reading from the same stale health cache — not independently verifying partition state.

Impact: the Redpanda Operator sidecar container uses the health endpoint for its readiness probe. When the health cache is stale, all broker pods show 1/2 (sidecar not ready), causing unnecessary alerts and operational confusion.

The sidecar logs confirm it is reacting to the stale health state:

{"level":"info","ts":"2026-05-06T14:26:48.267Z","logger":"Prober","msg":"broker has leaderless or under-replicated partitions","leaderless":0,"under-replicated":1}

All three broker pods (redpanda-0, redpanda-1, redpanda-2) show 1/2 simultaneously:

NAME         READY   STATUS    RESTARTS      AGE
redpanda-0   1/2     Running   0             11h
redpanda-1   1/2     Running   0             11h
redpanda-2   1/2     Running   1 (11h ago)   23h

What should have happened instead?

rpk cluster health and /v1/cluster/health_overview should reflect the actual partition state in real time, consistent with what /v1/partitions/kafka/<topic>/<partition> returns. If all partitions have valid leaders and all replicas are in sync, the cluster should report Healthy: true.

At minimum, the staleness window should be short (seconds, not tens of minutes), and there should be a mechanism to force a cache refresh.

How to reproduce the issue?

We observed this consistently after periods of produce/consume activity. A possible reproduction path:

1. Deploy a 3-broker Redpanda cluster (RF=3, SASL+TLS) via the Redpanda Operator on Kubernetes

2. Create a test topic with RF=3:

kubectl exec -n savannah-system redpanda-0 -c redpanda -- \
  rpk topic create test-grafana --partitions 3 --replicas 3

3. Produce a burst of ~30,000 messages:

kubectl exec -n savannah-system redpanda-0 -c redpanda -- \
  bash -c 'for i in $(seq 1 30000); do echo "message-$i payload-$(cat /dev/urandom | tr -dc a-z | head -c 200)"; done | rpk topic produce test-grafana --compression snappy'

4. Consume all messages with a named consumer group (triggers __consumer_offsets commit activity):

kubectl exec -n savannah-system redpanda-0 -c redpanda -- \
  rpk topic consume test-grafana --group test-grafana-consumer --num 30000

5. Observe rpk cluster health — it reports leaderless and under-replicated partitions across many topics (not just test-grafana):

kubectl exec -n savannah-system redpanda-0 -c redpanda -- rpk cluster health

6. At the same time, query individual partitions via the Admin API — they show valid leaders:

kubectl exec -n savannah-system redpanda-0 -c redpanda -- \
  curl -sk "https://localhost:9644/v1/partitions/kafka/owl.enriched.full/0" | python3 -m json.tool

7. Produce directly to one of the "leaderless" topics — it succeeds immediately with no error:

echo "health-test" | kubectl exec -i -n savannah-system redpanda-0 -c redpanda -- \
  rpk topic produce owl.enriched.full

The issue has occurred 3 times in 2 days on the same cluster, always with the same pattern.

Important: the stale health report includes partitions on topics that were never written to during the reproduce steps (e.g. fdb.trace-logs.v1, owl.enriched.full, pg.log_metadata.v2, k8s.events, mercury.v1). Only test-grafana was produced to. This rules out the possibility that the produce/consume activity directly caused those partitions to become unhealthy — the health cache appears to apply a cluster-wide stale view, not just to the partitions that were recently active.

Additional information

• All three observability surfaces share the same stale cache: rpk cluster health, /v1/cluster/health_overview, the Redpanda Console UI, and Prometheus metrics (redpanda_kafka_under_replicated_replicas). They all agree with each other but are all wrong simultaneously.
• The only authoritative source is /v1/partitions/kafka/<topic>/<partition>, which queries the Raft controller directly.
• The issue appears to be triggered by __consumer_offsets partition churn (burst of consumer group commits), though we cannot confirm this definitively.
• The cluster is Community Edition, no enterprise features enabled (partition_autobalancing_mode: node_add, core_balancing_continuous: false) even we are on the enterprise trial (but we explicitly disabled Enterprise features as we will use the Community edition).
• We did not attach the debug bundle as it contains internal topology information, but can provide it privately if needed.

Thank you for your help.

[dotnwat]

Thanks for the report. I've asked someone to take a look.

[AldoFusterTurpin]

Also, just to mention, rpk group list does not show all the consumer groups. It is missing test-grafana-consumer I believe it is not expected:

$ kubectl exec -n savannah-system redpanda-0 -c redpanda -- rpk group list
BROKER  GROUP

$ kubectl exec -n savannah-system redpanda-0 -c redpanda -- rpk group describe test-grafana-consumer
GROUP                  test-grafana-consumer
COORDINATOR-NODE       1
COORDINATOR-PARTITION  __consumer_offsets/7
STATE                  Empty
BALANCER
MEMBERS                0
TOTAL-LAG              0

TOPIC         PARTITION  CURRENT-OFFSET  LOG-START-OFFSET  LOG-END-OFFSET  LAG   MEMBER-ID  CLIENT-ID  HOST
test-grafana  0          11194           0                 11194           0
test-grafana  1          9020            0                 9020            0
test-grafana  2          9786            0                 9786            0

According to the --help it should show also empty groups.

kubectl exec -n savannah-system redpanda-0 -c redpanda -- rpk group list --help
List all groups.

This command lists all groups currently known to Redpanda, including empty
groups that have not yet expired...

[joe-redpanda]

@AldoFusterTurpin
Would it be possible to get logs spanning an instance of this issue?

[AldoFusterTurpin]

Thanks for the quick response.

FYI I just saw that there was a similar issue here, but in my case the problem has not been resolved.

You can find the logs below

kubectl  -n savannah-system logs redpanda-0 -c redpanda --since=6h > /tmp/redpanda-0.log
kubectl  -n savannah-system logs redpanda-1 -c redpanda --since=6h > /tmp/redpanda-1.log
kubectl -n savannah-system logs redpanda-2 -c redpanda --since=6h > /tmp/redpanda-2.log 

redpanda-0.log
redpanda-1.log
redpanda-2.log

[AldoFusterTurpin]

Not a C++ developer, but with the help of some AI, I tried to investigate a bit:

Summary of findings:
The stale health cache is a symptom, not the root cause. The real problem is:
1. redpanda-2 is consuming ~1133m CPU continuously vs ~160m for the other two brokers
2. This CPU saturation causes raftgen::append_entries_full_serde RPC calls from redpanda-0 to redpanda-2 to time out at 3000ms
3. Repeated timeouts trigger node_isolation_watcher → marks node as isolated
4. The health monitor collects stale partition reports → cache shows leaderless/under-replicated partitions
5. force_refresh::yes is never callable from outside — the cache has no external invalidation

and

Root cause chain:
1. redpanda-0 → redpanda-2 RPC timeouts (3000ms) on port 33145
2. node_status_table::is_isolated() checks last_seen for each peer — if ALL peers have last_seen > node_isolation_heartbeat_timeout (default 3000ms), node marks itself
isolated
3. node_isolation_watcher logs is_isolated: true, sets isolation on metadata cache
4. Health monitor collects stale reports (isolated node can't properly report partition state)
5. Cache stays stale because force_refresh::yes is never called from the API path

But it is strange because our config has (check below) and the only data we produced was a simple test producing 30K messages using rpk and consuming them using rpk

resources:
    cpu:
      cores: 4
    memory:
      enable_memory_locking: true
      container:
        max: 10Gi

so not sure why we have CPU starvation (we don't have any service connected to this cluster to produce or consume -yet-).

[AldoFusterTurpin]

Further diagnostics: CPU saturation on redpanda-2, shard-specific stale partition metadata

While waiting for the health cache issue to resolve, we did deeper diagnostics to provide more context. Here is what we found.

1. redpanda-2 is CPU-saturated

kubectl -n savannah-system top pods -l app.kubernetes.io/name=redpanda
NAME         CPU(cores)   MEMORY(bytes)
redpanda-0   168m         1550Mi
redpanda-1   161m         1509Mi
redpanda-2   1133m        2912Mi

redpanda-2 is consuming ~1133m CPU — roughly 7x more than the other two brokers, sustained for 15+ hours (the pod restarted once at 03:10 UTC and has been running since).

2. Reactor utilization above 1.0 on multiple shards

We queried the internal metrics endpoint on each broker:

kubectl -n savannah-system exec redpanda-2 -c redpanda -- \
curl -sk "https://localhost:9644/metrics" | grep "^vectorized_reactor_utilization\|^vectorized_reactor_polls"

Results:

redpanda-2:

vectorized_reactor_utilization{shard="0"} 1.794866
vectorized_reactor_utilization{shard="1"} 0.458119
vectorized_reactor_utilization{shard="2"} 0.955857
vectorized_reactor_utilization{shard="3"} 1.540884
vectorized_reactor_polls{shard="0"} 3801582831
vectorized_reactor_polls{shard="1"} 18396168757   ← anomalous: ~5x higher than other shards
vectorized_reactor_polls{shard="2"} 3324221984
vectorized_reactor_polls{shard="3"} 2823475168

redpanda-1 (also showing overload):

vectorized_reactor_utilization{shard="0"} 2.084223  ← >200%
vectorized_reactor_utilization{shard="2"} 0.977757

redpanda-0 (normal):

vectorized_reactor_utilization{shard="0"} ~normal
vectorized_reactor_polls{shard="*"} ~3.8B each (uniform)

Shard 1 on redpanda-2 has accumulated ~18.4B reactor polls vs ~3.8B on every shard of redpanda-0 — roughly 5x the expected rate, suggesting a busy-poll loop on that specific shard.

No stall detector events on any shard (vectorized_stall_detector_reported = 0).

3. The 6 "leaderless" partitions are all on shard 1 of redpanda-2

kubectl -n savannah-system exec redpanda-2 -c redpanda -- \
curl -sk "https://localhost:9644/v1/partitions" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for p in data:
    if p.get('leader', 0) == -1:
        print(f'{p[\"ns\"]}/{p[\"topic\"]}/{p[\"partition_id\"]} core={p[\"core\"]}')
"

Output:

kafka/mercury.v4/0        core=1
kafka/pg.log_metadata.v2/0  core=1
kafka/_schemas/0          core=1
kafka/fdb.trace-logs.v1/8  core=1
kafka/fdb.trace-logs.v1/6  core=1
kafka/fdb.trace-logs.v1/0  core=1

All 6 partitions showing leader=-1 are assigned to shard 1 of broker 2. No leaderless partitions appear on any other shard.

Cross-checking via individual partition endpoint (authoritative):

kubectl -n savannah-system exec redpanda-0 -c redpanda -- \
curl -sk "https://localhost:9644/v1/partitions/kafka/fdb.trace-logs.v1/0" | python3 -m json.tool | grep -E "leader_id|status"

"status": "done",
"leader_id": 0

All 6 partitions return valid leader_id and status: done from the individual endpoint — confirming it is shard 1's partition metadata cache on broker 2 that is stale, not a genuine leadership issue.

4. Partition leadership is heavily imbalanced

kubectl -n savannah-system exec redpanda-0 -c redpanda -- \
rpk cluster partitions list -a 2>/dev/null | awk 'NR>1 && $3 ~ /^[0-9]+$/ {count[$3]++} END {for (b in count) print "broker", b, "leads", count[b], "partitions"}' |
sort -k4 -rn

broker 0 leads 23 partitions
broker 2 leads 3 partitions
broker 1 leads 3 partitions

Broker 0 holds 23 partition leaders vs 3 each for brokers 1 and 2. This is likely a consequence of the stale health state (leadership not rebalancing while the cluster reports unhealthy), but we are noting it in case it is contributing to the CPU imbalance.

5. No active Raft recovery

kubectl -n savannah-system exec redpanda-2 -c redpanda -- \
curl -sk "https://localhost:9644/metrics" | grep "^vectorized_raft_recovery_partitions_to_recover\|^vectorized_raft_recovery_offsets_pending"

All zero on all shards — the CPU is not being consumed by catch-up replication after the restart.

Summary

	redpanda-0	redpanda-1	redpanda-2
CPU (kubectl top)	~168m	~161m	~1133m
Max reactor utilization	normal	2.08 (shard 0)	1.79 (shard 0)
Reactor polls shard 1	~3.8B	~3.1B	~18.4B
Partition leaders	23	3	3
Raft recovery pending	0	0	0
Stale partitions (leader=-1)	0	0	6 (all on shard 1)

The stale health reporting appears to be caused by a shard-level stale partition metadata cache on broker 2, shard 1, possibly related to the RPC timeouts we reported earlier (raftgen::append_entries_full_serde timeouts from broker 0 → broker 2 on port 33145). The CPU saturation and anomalous poll count on shard 1 may be a related symptom or contributing cause.

[AldoFusterTurpin]

@joe-redpanda Hello, an update after some hours (morning next day):

Root cause identified: memory leak in schema registry client retry path

After analyzing the crash logs from the previous container run (kubectl logs redpanda-2 -c redpanda --previous), the crash reason was explicitly recorded as OOM on shard 1, not a segfault: exit code 139 is a secondary effect of Seastar's abort handler faulting after memory was fully exhausted. The memory dump shows Used: 2G, Free: 0B with the top 10 allocation sites accounting for ~1.3GB, one of which explicitly includes libssl.so.3 in its stack.

The trigger is the internal schema_registry_client, which is misconfigured in our deployment, the operator renders redpanda.yaml with broker addresses and TLS for the schema registry client, but no sasl_mechanism, sasl_username, or sasl_password. Since port 9093 requires SASL auth, every connection attempt fails with sasl_authentication_failed [58]: Invalid credentials, visible in the logs from the very first second after broker startup, repeating every ~20 seconds for the entire 22-hour lifetime of the process.

We believe the retry path leaks TLS connection objects on each failed auth attempt without releasing them. ~3,960 retries × ~540KB per TLS object ≈ 2GB, which matches the shard budget exactly. We are fixing the missing credentials on our side.

However, we'd like to flag that a broker should not OOM due to failed auth retries, the connection objects should be released on failure regardless of whether authentication succeeds. Is there a known issue or fix for leaked TLS objects in the schema_registry_client retry path?

The issue was resolved and we'll fix the config from our side, but we believe there is something fishy you could check.

[StephanDollberg]

Can you share the memory dump (there is no PII in there). Will be easy for us to fix from that.

[AldoFusterTurpin]

Sure, here are the logs obtained with kubectl logs redpanda-2 -c redpanda --previous:

logs.txt

[AldoFusterTurpin]

Even I believe it is not the root cause of the cluster disruption, I created a PR with a small fix after I noticed something that could be related. Details here. Thank you.

[AldoFusterTurpin]

Hello, the problem of the outdated cache happened again after deploying a small producer/consumer application. What happened this time:

redpanda-0 was killed by Kubernetes and restarted. After restart it was behind on the Raft log for several partitions, causing the cluster to temporarily report 2 unavailable partitions and 7 under-replicated replicas. The cluster recovered on its own once redpanda-0 caught up, but the disruption was visible in Redpanda Console.

This happened a few hours after deploying a program that was reading and writing to a test topic.

The Console showed:

• Unavailable Partitions: 2 (red)
• Under-Replicated Replicas: 7 (red)
• Uptime (oldest broker): 32.7 mins — a broker had just restarted

kubectl get pods -n savannah-system -l app.kubernetes.io/name=redpanda

Output:

NAME         READY   STATUS    RESTARTS      AGE
redpanda-0   1/2     Running   1 (33m ago)   5d15h
redpanda-1   1/2     Running   0             5d15h
redpanda-2   1/2     Running   0             4d4h

redpanda-0 had 1 restart ~33 minutes ago. The 1/2 ready state is normal during recovery (operator sidecar marks not-ready until partitions are healthy).

kubectl get events -n savannah-system --sort-by='.lastTimestamp' | grep -i "redpanda\|OOMKill\|kill\|restart\|evict\|backoff" | tail -20

Key output:

Warning  Unhealthy   40m (x426 over 3h5m)  kubelet  Readiness probe failed: Get "http://...:8093/healthz": context deadline exceeded
Warning  Unhealthy   35m (x2 over 35m)     kubelet  Liveness probe failed: dial tcp ...:9644: i/o timeout
Normal   Killing     35m                   kubelet  Container redpanda failed liveness probe, will be restarted

The liveness probe timed out trying to open a TCP connection to port 9644 (admin API). After 3 consecutive failures (3 × 1s timeout = 3 seconds), Kubernetes killed and restarted the container.

kubectl get pod redpanda-0 -n savannah-system -o json

{
  "failureThreshold": 3,
  "initialDelaySeconds": 10,
  "periodSeconds": 10,
  "successThreshold": 1,
  "tcpSocket": {
    "port": 9644
  },
  "timeoutSeconds": 1
}

timeoutSeconds: 1 with failureThreshold: 3 means the broker is killed if port 9644 is unresponsive for just 3 seconds total. Under producer/consumer load the admin API can briefly stall, which is enough to cross this threshold.

I believe this is a false positive kill: the broker was healthy and serving Kafka traffic, but kubelet killed it because it couldn't open a TCP socket fast enough under load.

Even we changed the liveness probe config and got applied:

 k get statefulsets.apps redpanda -o yaml | rg livenessProbe -A 7
        livenessProbe:
          failureThreshold: 6
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 9644
          timeoutSeconds: 5

we are hitting still the cache issue where the general rpk health shows

kubectl exec -n savannah-system redpanda-0 -c redpanda -- rpk cluster health
Healthy:                          false
Unhealthy reasons:                [leaderless_partitions under_replicated_partitions]
. . .
Leaderless partitions (12):       [. . .]

but when you query directly using the /v1 endpoint it tells it has in fact a leader.

kubectl exec redpanda-0 -n savannah-system -c redpanda -- curl -sk https://localhost:9644/v1/partitions/kafka/k8s.nodes/0
{"ns": "kafka", "topic": "k8s.nodes", "partition_id": 0, "status": "done", "leader_id": 2, "raft_group_id": 46, "replicas": [{"node_id": 2, "core": 0},{"node_id": 1, "core": 0},{"node_id": 0, "core": 0}], "disabled": false}

Also, all the topics show:

k get topic
NAME                         READY   STATUS
[topic_name]            False      Topic reconciliation failed

[dotnwat]

Thanks @AldoFusterTurpin we do have this on our backlog. Don't hestitate to post more updates and information to this issue.