Fail to move cluster with associated IPAddresses

[mcbenjemaa]

What steps did you take and what happened?

I was trying to move a cluster from a bootstrap cluster to the new mgmt-cluster, which failed, because of the webhook.

logs:

Creating IPAddress="mgmt-cluster-control-plane-9p4r4" Namespace="default"
Object already exists, updating IPAddress="mgmt-cluster-control-plane-9p4r4" Namespace="default"
Retrying with backoff Cause="error updating \"ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress\" default/mgmt-cluster-control-plane-9p4r4: admission webhook \"validation.ipaddress.ipam.cluster.x-k8s.io\" denied the request: spec: Forbidden: the spec of IPAddress is immutable

What did you expect to happen?

I expect the clusterctl move succeed

Cluster API version

v1.5.1

Kubernetes version

v1.26.7

Anything else you would like to add?

Error: [action failed after 10 attempts: error updating "ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress" default/mgmt-cluster-control-plane-pwhfv: admission webhook "validation.ipaddress.ipam.cluster.x-k8s.io" denied the request: spec: Forbidden: the spec of IPAddress is immutable, action failed after 10 attempts: error updating "ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress" default/mgmt-cluster-control-plane-6xk8h: admission webhook "validation.ipaddress.ipam.cluster.x-k8s.io" denied the request: spec: Forbidden: the spec of IPAddress is immutable, action failed after 10 attempts: error updating "ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress" default/mgmt-cluster-control-plane-9p4r4: admission webhook "validation.ipaddress.ipam.cluster.x-k8s.io" denied the request: spec: Forbidden: the spec of IPAddress is immutable]

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[killianmuldoon]

/triage accepted

It looks like an IPAddress object with the same name already exists in your target cluster. Do you know how it got there? Did you try to run move before?

[mcbenjemaa]

because the clusterctl move actually moves the IPAddressClaim before IPAddress,
and that makes it fail.

I guess the fix would be to ignore IPAddresses as they will re-created by the controller via IPAddressClaim.

[killianmuldoon]

We should definitely fix this up if there's a specific order expected.

@schrej - do you have any opinions on a good solution here?

[schrej]

For the in-cluster provider we'll definitely have to move the IPAddresses. Otherwise new addresses will be assigned in a random order, since the in-cluster provider, as the name suggests, only stores the state within the API. For providers interacting with external IPAM systems this is less of an issue, depending on implementation.

Unfortunately I'm not very familiar with clusterctl move, so I'll probably need to learn a bit before I can give a good answer, and I don't know when I'll have time to do so, sorry.
If the order can be influenced that might be the easiest solution. If there are annotations present for moved resources we could add exceptions to validation.

[mcbenjemaa]

If we have to move them, I guess IPAddresses must be moved first, and then the claims.

[killianmuldoon]

Thanks for the input @schrej - sounds like we might need to handle this case in clusterctl move.

One question on:

because the clusterctl move actually moves the IPAddressClaim before IPAddress, and that makes it fail.

How does this cause the move to fail? The objects should be moved independently one after another and reconciliation should be paused. What's causing the spec to be mutated in this case?

[mcbenjemaa]

How does this cause the move to fail? The objects should be moved independently one after another and reconciliation should be paused. What's causing the spec to be mutated in this case?

Probably because reconciliation for IPAddressClaim is not paused

[killianmuldoon]

Probably because reconciliation for IPAddressClaim is not paused

Oh that's a bug then - did it get the annotation?

[mcbenjemaa]

I have to recreate the cluster and try it again.

[killianmuldoon]

I have to recreate the cluster and try it again.

Thank you! Just out of interest what infrastructure provider are you using (and what version)? I guess this issue could be in the reconciler not correctly processing a paused annotation.

[mcbenjemaa]

I'm building a provider for Proxmox, and I'm injecting static IPs with IPAM incluster.