The Console controller renders its config from a RenderState built with nil node pools, so
kafka.brokers and schemaRegistry.urls come out empty and are dropped by omitempty. Console
then rejects its own config at startup and CrashLoopBackOffs.
This is reachable by following the documented NodePool migration procedure, which requires
spec.clusterSpec.statefulset.replicas: 0.
Environment
|
|
| Operator |
v26.2.2 (call sites byte-identical at v26.2.1 and v26.2.3) |
| Console image |
v3.x |
| CR |
standalone Console CR (spec.clusterSpec.console.enabled: false on the Redpanda CR) |
Root cause
ConvertV2ToRenderState(config, defaulters, cluster, pools)
(operator/api/redpanda/v1alpha2/conversion/to_render.go:36) takes node pools as its fourth
argument. Two call sites pass nil:
operator/internal/controller/console/controller.go:569
operator/internal/controller/pipeline/cluster.go:351
The three call sites that get it right pass cluster.NodePools:
operator/internal/lifecycle/v2_simple_resources.go:59
operator/internal/lifecycle/v2_simple_resources.go:145
operator/internal/lifecycle/v2_node_pools.go:60
With nil pools, state.Pools is empty, so BrokerList() iterates i < Replicas where
Replicas is 0 and returns a nil slice. omitempty
(charts/console/zz_generated.config.go:57) then drops the key from the rendered ConfigMap
entirely.
Steps to reproduce
Redpanda CR (v1alpha2) with brokers served by NodePool CRs and
spec.clusterSpec.statefulset.replicas: 0 — the documented post-migration state.
spec.clusterSpec.console.enabled: false, with a standalone Console CR referencing the
cluster.
- Let the Console controller reconcile and inspect the rendered ConfigMap.
Observed
Rendered ConfigMap: kafka: contains only sasl and tls — no brokers. schemaRegistry:
contains only authentication and enabled — no urls. redpanda.adminApi.urls survives,
because only the statically enumerated stanzas break; the Admin/Topic/User/Schema clients resolve
via Service/SRV discovery.
Console container log:
{"level":"ERROR","msg":"failed to load yaml config","error":"config validation failed:
failed to validate API config: failed to validate Kafka config:
you must specify at least one broker to connect to"}
The pod exits code 1 with startedAt == finishedAt — immediate config rejection, not a
connection timeout — and enters CrashLoopBackOff. The previous ReplicaSet's pod stays Ready, so
the outage only surfaces on the next Console rollout.
Expected
Console's rendered config should enumerate the brokers from the cluster's NodePools, exactly as
the lifecycle call sites do.
Suggested fix
Pass the cluster's NodePools at both call sites.
Separately, the Console controller does not watch NodePool, so it will not re-reconcile when a
pool is created or resized even once the nil is fixed. A pool resize will leave Console's broker
list stale until something else triggers a reconcile.
Scope
pipeline/cluster.go:351 feeds AsStaticConfigSource() for Redpanda Connect Pipeline CRs, so
pipelines resolving their Kafka connection details from a v2 cluster in the same state should be
affected the same way. I have not exercised that path directly — it is a code-read only.
The Console controller renders its config from a
RenderStatebuilt withnilnode pools, sokafka.brokersandschemaRegistry.urlscome out empty and are dropped byomitempty. Consolethen rejects its own config at startup and CrashLoopBackOffs.
This is reachable by following the documented NodePool migration procedure, which requires
spec.clusterSpec.statefulset.replicas: 0.Environment
v26.2.2(call sites byte-identical atv26.2.1andv26.2.3)v3.xConsoleCR (spec.clusterSpec.console.enabled: falseon theRedpandaCR)Root cause
ConvertV2ToRenderState(config, defaulters, cluster, pools)(
operator/api/redpanda/v1alpha2/conversion/to_render.go:36) takes node pools as its fourthargument. Two call sites pass
nil:operator/internal/controller/console/controller.go:569operator/internal/controller/pipeline/cluster.go:351The three call sites that get it right pass
cluster.NodePools:operator/internal/lifecycle/v2_simple_resources.go:59operator/internal/lifecycle/v2_simple_resources.go:145operator/internal/lifecycle/v2_node_pools.go:60With
nilpools,state.Poolsis empty, soBrokerList()iteratesi < ReplicaswhereReplicasis 0 and returns a nil slice.omitempty(
charts/console/zz_generated.config.go:57) then drops the key from the rendered ConfigMapentirely.
Steps to reproduce
RedpandaCR (v1alpha2) with brokers served byNodePoolCRs andspec.clusterSpec.statefulset.replicas: 0— the documented post-migration state.spec.clusterSpec.console.enabled: false, with a standaloneConsoleCR referencing thecluster.
Observed
Rendered ConfigMap:
kafka:contains onlysaslandtls— nobrokers.schemaRegistry:contains only
authenticationandenabled— nourls.redpanda.adminApi.urlssurvives,because only the statically enumerated stanzas break; the Admin/Topic/User/Schema clients resolve
via Service/SRV discovery.
Console container log:
The pod exits code 1 with
startedAt == finishedAt— immediate config rejection, not aconnection timeout — and enters CrashLoopBackOff. The previous ReplicaSet's pod stays Ready, so
the outage only surfaces on the next Console rollout.
Expected
Console's rendered config should enumerate the brokers from the cluster's NodePools, exactly as
the
lifecyclecall sites do.Suggested fix
Pass the cluster's NodePools at both call sites.
Separately, the Console controller does not watch
NodePool, so it will not re-reconcile when apool is created or resized even once the
nilis fixed. A pool resize will leave Console's brokerlist stale until something else triggers a reconcile.
Scope
pipeline/cluster.go:351feedsAsStaticConfigSource()for Redpanda ConnectPipelineCRs, sopipelines resolving their Kafka connection details from a v2 cluster in the same state should be
affected the same way. I have not exercised that path directly — it is a code-read only.