When setup-gcp bootstrap (or create cluster) finds that the cluster already exists, its reconcile path deletes the cluster and creates a new one if any of three settings differ from the tool's config (cluster.go#L161-L190):
- the network (
projects/<PROJECT_ID>/global/networks/<NETWORK>);
- the subnetwork (
projects/<PROJECT_ID>/regions/<GCE_REGION>/subnetworks/<SUBNETWORK>);
- Dataplane V2 on or off.
There's no prompt and no flag guarding it. The log line before the delete is INFO Mismatch in subnetwork. Everything running on the cluster is lost.
Why it's easy to hit
bootstrap expects Dataplane V2 without exposing the setting. Only create cluster registers --enable-dataplane-v2 (L403). But the two commands share the global cfg, and that registration sets cfg.EnableDataplaneV2 = getEnv("ENABLE_DATAPLANE_V2", true) during init(). So bootstrap silently expects Dataplane V2. I confirmed it with a throwaway test against the tree: bootstrap sees EnableDataplaneV2 == true. GKE Standard clusters use the legacy datapath unless created with --enable-dataplane-v2, so any such cluster passed to bootstrap is deleted.
- The subnetwork defaults to
default. GKE now commonly creates a per-cluster subnet named gke-<cluster>-subnet-<hash>. In one project I checked (13 clusters, a mix of Standard and Autopilot), 7 clusters are on a subnet other than default, and 2 more are on the legacy datapath. Running bootstrap with default settings against any of those 9 would delete and recreate it.
- Shared VPC can never match. The expected network is built from
PROJECT_ID. A cluster in a service project whose network lives in the host project reports projects/<HOST>/global/networks/..., so the suffix check fails whatever NETWORK is set to.
Proposed fix
- Never delete an existing cluster implicitly. On a mismatch, stop with an error that names the setting and both values, and say how to proceed: recreate it yourself, or use a different cluster name. If automatic recreation is wanted for dev loops, put it behind an explicit flag such as
--recreate-on-mismatch.
- Register
--enable-dataplane-v2 / ENABLE_DATAPLANE_V2 on bootstrap too, so its expectation is visible and can be overridden. Better still, only compare network, subnetwork and datapath when the caller set them explicitly.
The GKE installer runs bootstrap against existing clusters, so I'm adding a guard there now, passing the cluster's actual network, subnetwork and datapath so the reconcile finds nothing to recreate. An upstream fix would protect everyone else who runs bootstrap against an existing cluster. Happy to send the PR.
When
setup-gcp bootstrap(orcreate cluster) finds that the cluster already exists, its reconcile path deletes the cluster and creates a new one if any of three settings differ from the tool's config (cluster.go#L161-L190):projects/<PROJECT_ID>/global/networks/<NETWORK>);projects/<PROJECT_ID>/regions/<GCE_REGION>/subnetworks/<SUBNETWORK>);There's no prompt and no flag guarding it. The log line before the delete is
INFO Mismatch in subnetwork. Everything running on the cluster is lost.Why it's easy to hit
bootstrapexpects Dataplane V2 without exposing the setting. Onlycreate clusterregisters--enable-dataplane-v2(L403). But the two commands share the globalcfg, and that registration setscfg.EnableDataplaneV2 = getEnv("ENABLE_DATAPLANE_V2", true)duringinit(). Sobootstrapsilently expects Dataplane V2. I confirmed it with a throwaway test against the tree:bootstrapseesEnableDataplaneV2 == true. GKE Standard clusters use the legacy datapath unless created with--enable-dataplane-v2, so any such cluster passed tobootstrapis deleted.default. GKE now commonly creates a per-cluster subnet namedgke-<cluster>-subnet-<hash>. In one project I checked (13 clusters, a mix of Standard and Autopilot), 7 clusters are on a subnet other thandefault, and 2 more are on the legacy datapath. Runningbootstrapwith default settings against any of those 9 would delete and recreate it.PROJECT_ID. A cluster in a service project whose network lives in the host project reportsprojects/<HOST>/global/networks/..., so the suffix check fails whateverNETWORKis set to.Proposed fix
--recreate-on-mismatch.--enable-dataplane-v2/ENABLE_DATAPLANE_V2onbootstraptoo, so its expectation is visible and can be overridden. Better still, only compare network, subnetwork and datapath when the caller set them explicitly.The GKE installer runs
bootstrapagainst existing clusters, so I'm adding a guard there now, passing the cluster's actual network, subnetwork and datapath so the reconcile finds nothing to recreate. An upstream fix would protect everyone else who runsbootstrapagainst an existing cluster. Happy to send the PR.