fix: wait for rook ceph migrator pod ready

[emosbaugh]

What this PR does / why we need it:

We do two migrations back to back. This code has a race condition and will try to wait for the previous pod that is terminating rather than the pod from the new deployment.

https://testgrid.kurl.sh/run/STAGING-daily-storage-migration-ee0da9b-2023-05-25T01:28:20Z?kurlLogsInstanceId=vvcexmochqzfsqvy&nodeId=vvcexmochqzfsqvy-initialprimary#L0

2023-05-25 12:29:56+00:00 ✔ Rook Flex volumes to CSI volumes migrated successfully
2023-05-25 12:29:56+00:00 storageclass.storage.k8s.io "default" deleted
2023-05-25 12:29:57+00:00 storageclass.storage.k8s.io/default created
2023-05-25 12:29:57+00:00 cephblockpool.ceph.rook.io/replicapool unchanged
2023-05-25 12:29:57+00:00 ⚙  Migrating Rook Flex volumes to CSI volumes
2023-05-25 12:29:57+00:00 + ./bin/kurl rook flexvolume-to-csi --source-sc rook-ceph-tmp --destination-sc default --node vvcexmochqzfsqvy-initialprimary --pv-migrator-bin-path /var/lib/kurl/bin/rook-pv-migrator --ceph-migrator-image rook/ceph:v1.7.11
2023-05-25 12:29:58+00:00 Running rook-ceph-migrator deployment ...
2023-05-25 12:29:58+00:00 # Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2023-05-25 12:29:58+00:00 Waiting for rook-ceph-migrator deployment to be ready ...
2023-05-25 12:30:08+00:00 Deleting flex migrator ...
2023-05-25 12:30:08+00:00 Deleted flex migrator
2023-05-25 12:30:08+00:00 Error: wait for flex migrator pod: wait for rook-ceph-migrator pod: pods "rook-ceph-migrator-8c89874c6-77hvf" not found

Which issue(s) this PR fixes:

Fixes NONE

Special notes for your reviewer:

Steps to reproduce

Does this PR introduce a user-facing change?

Fixes an issue that could cause Rook upgrades from version 1.0.4 to 1.7.x to fail with error rook-ceph-migrator pod not found.

Does this PR require documentation?

NONE

[laverya]

I think you need to have something after this that handles "no pod was ready"

after all, that's why we initially had the call to WaitForPodReady

[laverya]

seems like this might be covered already by WaitForDeploymentReady

[emosbaugh]

i believe waitfordeploymentready above will wait for one replica (pod) to be ready