Continuous running rsync loop when trigger is always

[dheerajjoshim]

In the current rsync-tls (and rsync-ssh) mover implementation, the source mover pod is terminated once the data transfer is completed. And depending on the trigger, the mover pod will be recreated immediately (trigger always) or at a specific time (trigger schedule).

For a huge number of PVCs getting synced to the remote data centre, this causes some issues.

• The termination of the pod puts some load on the API server (Pod scheduled and terminated). Though this was not huge, in a massive data centre with many teams replicating the volumes leads to a situation where some load is observed on the K8S api server
• It tries to pull the mover pod image from the repo every time the pod is created, thus increasing the request to the repository
• Pod deletion/creation causes alerts to be generated. While this is configurable in our alerting rule, finding the optimal number is tricky when the teams are scheduling the replication at different times.

Is it a wise idea to change the always trigger to run the rsync in a loop within the mover pod and only terminate and recreate the pod when the rsync fails? This will make the source mover pod run for a long time till the failure occurs or the ReplicationSource custom resource is explicitly deleted, thus minimising the above-mentioned issues.

[tesshuflower]

This is kind of down to the nature of kubernetes - the movers are run in jobs, and that allows the volsync controller to coordinate doing things like snapshotting, scheduling etc before/after the job completes (or retrying if it fails).

As an example, the replicationdestination is one where it's common in rsync/rsync-tls movers to run this continuously - but the job will still complete after a replication is successful. This enables the volsync controller to know that the replicationdestination has completed, and proceed to take a destination snapshot (without the mover unintentionally writing anything to the disk).

When it comes to pulling images from the repo, volsync isn't setting a specific imagePullPolicy on the job, which means it should probably be defaulting to IfNotPresent which means it shouldn't pull a new image if it already exists on the node.