Skip to content

setup-gcp: bootstrap deletes and recreates an existing cluster when its subnetwork, network, or Dataplane V2 setting differs #2341

Description

When setup-gcp bootstrap (or create cluster) finds that the cluster already exists, its reconcile path deletes the cluster and creates a new one if any of three settings differ from the tool's config (cluster.go#L161-L190):

  • the network (projects/<PROJECT_ID>/global/networks/<NETWORK>);
  • the subnetwork (projects/<PROJECT_ID>/regions/<GCE_REGION>/subnetworks/<SUBNETWORK>);
  • Dataplane V2 on or off.

There's no prompt and no flag guarding it. The log line before the delete is INFO Mismatch in subnetwork. Everything running on the cluster is lost.

Why it's easy to hit

  1. bootstrap expects Dataplane V2 without exposing the setting. Only create cluster registers --enable-dataplane-v2 (L403). But the two commands share the global cfg, and that registration sets cfg.EnableDataplaneV2 = getEnv("ENABLE_DATAPLANE_V2", true) during init(). So bootstrap silently expects Dataplane V2. I confirmed it with a throwaway test against the tree: bootstrap sees EnableDataplaneV2 == true. GKE Standard clusters use the legacy datapath unless created with --enable-dataplane-v2, so any such cluster passed to bootstrap is deleted.
  2. The subnetwork defaults to default. GKE now commonly creates a per-cluster subnet named gke-<cluster>-subnet-<hash>. In one project I checked (13 clusters, a mix of Standard and Autopilot), 7 clusters are on a subnet other than default, and 2 more are on the legacy datapath. Running bootstrap with default settings against any of those 9 would delete and recreate it.
  3. Shared VPC can never match. The expected network is built from PROJECT_ID. A cluster in a service project whose network lives in the host project reports projects/<HOST>/global/networks/..., so the suffix check fails whatever NETWORK is set to.

Proposed fix

  • Never delete an existing cluster implicitly. On a mismatch, stop with an error that names the setting and both values, and say how to proceed: recreate it yourself, or use a different cluster name. If automatic recreation is wanted for dev loops, put it behind an explicit flag such as --recreate-on-mismatch.
  • Register --enable-dataplane-v2 / ENABLE_DATAPLANE_V2 on bootstrap too, so its expectation is visible and can be overridden. Better still, only compare network, subnetwork and datapath when the caller set them explicitly.

The GKE installer runs bootstrap against existing clusters, so I'm adding a guard there now, passing the cluster's actual network, subnetwork and datapath so the reconcile finds nothing to recreate. An upstream fix would protect everyone else who runs bootstrap against an existing cluster. Happy to send the PR.

Activity

  1. mastersingh24 commented on Oct 8, 2026

    @mastersingh24
    Author

    Reproduced on a throwaway GKE cluster. It was created with a GKE-made subnet (--create-subnetwork "") and left on the legacy datapath, which are the defaults for many Standard clusters. I ran setup-gcp create cluster at 756c2a5 with SUBNETWORK=default; bootstrap runs the same reconcile at step 2/7. It deleted the cluster without a prompt:

    INFO Cluster exists. Checking attributes... cluster=net-guard-probe
    INFO Mismatch in subnetwork current=projects/<project>/regions/us-central1/subnetworks/gke-net-guard-probe-subnet-b2563343 expected=projects/<project>/regions/us-central1/subnetworks/default
    INFO Deleting cluster cluster=net-guard-probe
    

    With NETWORK, SUBNETWORK and ENABLE_DATAPLANE_V2 set to the cluster's own values, the same reconcile passed all three checks and moved on to Workload Identity. That's the workaround the GKE installer is adopting in ai-on-gke/substrate-gke for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions