Skip to content

Console and Pipeline controllers pass nil NodePools to ConvertV2ToRenderState, rendering Console with an empty broker list #1907

Description

@rjustice-rp

The Console controller renders its config from a RenderState built with nil node pools, so
kafka.brokers and schemaRegistry.urls come out empty and are dropped by omitempty. Console
then rejects its own config at startup and CrashLoopBackOffs.

This is reachable by following the documented NodePool migration procedure, which requires
spec.clusterSpec.statefulset.replicas: 0.

Environment

Operator v26.2.2 (call sites byte-identical at v26.2.1 and v26.2.3)
Console image v3.x
CR standalone Console CR (spec.clusterSpec.console.enabled: false on the Redpanda CR)

Root cause

ConvertV2ToRenderState(config, defaulters, cluster, pools)
(operator/api/redpanda/v1alpha2/conversion/to_render.go:36) takes node pools as its fourth
argument. Two call sites pass nil:

  • operator/internal/controller/console/controller.go:569
  • operator/internal/controller/pipeline/cluster.go:351

The three call sites that get it right pass cluster.NodePools:

  • operator/internal/lifecycle/v2_simple_resources.go:59
  • operator/internal/lifecycle/v2_simple_resources.go:145
  • operator/internal/lifecycle/v2_node_pools.go:60

With nil pools, state.Pools is empty, so BrokerList() iterates i < Replicas where
Replicas is 0 and returns a nil slice. omitempty
(charts/console/zz_generated.config.go:57) then drops the key from the rendered ConfigMap
entirely.

Steps to reproduce

  1. Redpanda CR (v1alpha2) with brokers served by NodePool CRs and
    spec.clusterSpec.statefulset.replicas: 0 — the documented post-migration state.
  2. spec.clusterSpec.console.enabled: false, with a standalone Console CR referencing the
    cluster.
  3. Let the Console controller reconcile and inspect the rendered ConfigMap.

Observed

Rendered ConfigMap: kafka: contains only sasl and tls — no brokers. schemaRegistry:
contains only authentication and enabled — no urls. redpanda.adminApi.urls survives,
because only the statically enumerated stanzas break; the Admin/Topic/User/Schema clients resolve
via Service/SRV discovery.

Console container log:

{"level":"ERROR","msg":"failed to load yaml config","error":"config validation failed:
failed to validate API config: failed to validate Kafka config:
you must specify at least one broker to connect to"}

The pod exits code 1 with startedAt == finishedAt — immediate config rejection, not a
connection timeout — and enters CrashLoopBackOff. The previous ReplicaSet's pod stays Ready, so
the outage only surfaces on the next Console rollout.

Expected

Console's rendered config should enumerate the brokers from the cluster's NodePools, exactly as
the lifecycle call sites do.

Suggested fix

Pass the cluster's NodePools at both call sites.

Separately, the Console controller does not watch NodePool, so it will not re-reconcile when a
pool is created or resized even once the nil is fixed. A pool resize will leave Console's broker
list stale until something else triggers a reconcile.

Scope

pipeline/cluster.go:351 feeds AsStaticConfigSource() for Redpanda Connect Pipeline CRs, so
pipelines resolving their Kafka connection details from a v2 cluster in the same state should be
affected the same way. I have not exercised that path directly — it is a code-read only.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions