CI: PowerVS cluster creation fails with no fixed IP addresses available

[Amulyam24]

In CI, we create 2 clusters as a part of the test suite. After the first cluster is created successfully, during the the second cluster deployment, first machine creation fails with

"message":"Build of instance 70daa327-9052-4168-90ea-27677aac45bc aborted: Failed to allocate the network(s) with error No fixed IP addresses available for network: 8b7e4d36-7142-4216-83d1-4a4849c229c0, not rescheduling."

This is due to improper cleanup of VMs created as a part of the first cluster deployment.

{
  "pvmInstances": [
    {
      "addresses": [],
      "creationDate": "2024-07-08T21:46:35.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.427994",
        "reason": "PENDING",
        "status": "PENDING"
      },
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/70daa327-9052-4168-90ea-27677aac45bc",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "memory": 0,
      "networks": [],
      "osType": "",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "",
      "processors": 0,
      "pvmInstanceID": "70daa327-9052-4168-90ea-27677aac45bc",
      "serverName": "capibm-e2e-ri4t2i-control-plane-4dl6k",
      "srcs": [],
      "status": "BUILD",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "spawning",
      "updatedDate": "2024-07-08T21:46:35.000Z",
      "virtualCores": {
        "assigned": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89/networks/capi-net-CzwSz",
          "ip": "192.168.157.165",
          "ipAddress": "192.168.157.165",
          "macAddress": "fa:7b:e2:cc:61:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:36:00.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428057",
        "reason": "rmc_state - inactive",
        "status": "WARNING"
      },
      "hostID": 4,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89/networks/capi-net-CzwSz",
          "ip": "192.168.157.165",
          "ipAddress": "192.168.157.165",
          "macAddress": "fa:7b:e2:cc:61:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Unknown",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "5fbf4b71-caad-4e49-9541-bff8b6d6af89",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-sbcrc",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:36:00.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60/networks/capi-net-CzwSz",
          "ip": "192.168.157.163",
          "ipAddress": "192.168.157.163",
          "macAddress": "fa:63:9b:99:6b:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:35:48.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428106",
        "reason": "rmc_state - inactive",
        "status": "WARNING"
      },
      "hostID": 29,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60/networks/capi-net-CzwSz",
          "ip": "192.168.157.163",
          "ipAddress": "192.168.157.163",
          "macAddress": "fa:63:9b:99:6b:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "936e6efa-05d4-47dd-ba6c-fca24240bc60",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-4mf49",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:35:48.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d/networks/capi-net-CzwSz",
          "ip": "192.168.157.162",
          "ipAddress": "192.168.157.162",
          "macAddress": "fa:19:94:45:15:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:25:49.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428142",
        "status": "OK"
      },
      "hostID": 4,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d/networks/capi-net-CzwSz",
          "ip": "192.168.157.162",
          "ipAddress": "192.168.157.162",
          "macAddress": "fa:19:94:45:15:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "4d7910d6-abe7-4a17-ad30-ea27b46e987d",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-crx6p",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:25:49.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f/networks/capi-net-CzwSz",
          "ip": "192.168.157.164",
          "ipAddress": "192.168.157.164",
          "macAddress": "fa:fa:b2:69:da:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:16:45.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428172",
        "status": "OK"
      },
      "hostID": 29,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f/networks/capi-net-CzwSz",
          "ip": "192.168.157.164",
          "ipAddress": "192.168.157.164",
          "macAddress": "fa:fa:b2:69:da:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "316dc448-77ac-4bdb-8342-7605d95b9c9f",
      "serverName": "capibm-e2e-jb3h95-control-plane-4w5sw",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:16:45.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    }
  ]
}

Job link - https://prow.ppc64le-cloud.cis.ibm.net/view/gs/ppc64le-kubernetes/logs/periodic-capi-provider-ibmcloud-e2e-powervs/1810418740110561280

[mkumatag]

is this happening because vm deleting is taking some time and not waiting for vm to be disappear?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned