Migration fails to create NetCom PropsContainers due to API rate limits

[stackblaze-adam]

Migration fails to create NetCom PropsContainers due to API rate limits

I'm running Piraeus Operator v2.10.3 with LINSTOR 1.32.3 on a fresh K3s cluster. When the migration runs (specifically step 4→5 that renames netcom to NetCom), it tries to create a bunch of NetCom PropsContainers via the Kubernetes API.

The problem is when it hits Kubernetes API rate limits (429 errors), some of those property creations fail, but the migration just continues anyway and marks itself as "Completed". So you end up with only 8 out of 24 NetCom properties created, and the controller can't start because it's missing critical ones like NetCom/PlainConnector/Enabled, NetCom/PlainConnector/Port, and NetCom/PlainConnector/Type.

I can see in the migration logs that it's getting 429s:

14:40:45.384 [vert.x-eventloop-thread-1] DEBUG io.fabric8.kubernetes.client.http.StandardHttpClient -- HTTP operation on url: https://10.43.0.1:443/apis/internal.linstor.linbit.com/v1-19-1/ebsremotes should be retried as the response code was 429, retrying after 1000 millis

But the migration completes anyway. When I check what PropsContainers actually got created:

kubectl get propscontainers -o json | jq -r '.items[] | select(.spec.prop_key | startswith("NetCom/")) | .spec.prop_key' | wc -l
8  # Should be 24

The missing ones are the operational properties needed for PlainConnector/SslConnector to actually work. Without them, the controller fails with "The configuration entry 'NetCom/PlainConnector/Port' is missing" errors.

I had to manually create the missing PropsContainers to get it working. This seems like the migration should either:

1. Retry failed property creations when it gets 429s
2. Actually fail if it can't create all the required properties
3. At least warn that some properties are missing

Right now it silently succeeds but leaves the cluster in a broken state.

Environment:

• Piraeus Operator: v2.10.3 (latest)
• LINSTOR: 1.32.3
• K3s: v1.31.12+k3s1
• Kubernetes API rate limits: default (400 read, 200 write)

[WanzenBug]

This seems to be an issue for https://github.com/LINBIT/linstor-server.

It seems, there is already a retry mechanism with the Kubernetes Client in use, but it seems to default to 3 retries only. Then, the request should fail. Perhaps there is a bug in that logic during migrations 🤔