Add continuous transfer mode

[ig11987]

Closes: #16

[ruuda]

Ooh, nice! I didn’t review in depth yet, but one thing that comes to mind, our goal for this is to sync a directory, where the sending side may still be producing new files, right? So we sync once, but by the time it’s complete, there may be new files, so we do it again, etc., until the difference is very small, then we stop the process that’s writing files, we do a final sync, and at that point we know the copy is complete.

With the current implementation, would we need to execute fastsync for multiple iterations? Because if this is the goal anyway, and if the receiver and sender negotiate anyway, then maybe it makes sense to build that loop right into fastsync itself, and make the protocol more interactive. After the sender sends its last piece, it could re-scan its input files to see if any of them changed, and if so go for a new round. Not sure if that would be a major complication though. What do you think? I don’t mean to scope-creep things, we can also leave it for a later version, but if it’s a few-line modification I think that could improve the practical usability a lot.

[ig11987]

With the current implementation, would we need to execute fastsync for multiple iterations?

In my use-case it's 2 times.
a: fastsync without incremental (target directory is empty, so does not matter if it gets deleted before accepting packages - current behavior). (1)
b: Then stop the workload that produces the files that are transferring
c: Sync with incremental for any leftover files. (2)

So we sync once, but by the time it’s complete, there may be new files, so we do it again, etc., until the difference is very small, then we stop the process that’s writing files, we do a final sync, and at that point we know the copy is complete.

The transfer is usually much faster than the speed of producing files, so I have not observed this being very useful for the workloads I have worked with so far.
But I do see the potential of continuous mode being useful.

There is are some potential risks:

1. the process never ending - we can offload this to the user
2. how to deal with deletes on sender (I don't think it's currently handled) - could produce many files piling up on the receiver.
3. the code becoming too complicated for the added utility when compared to incremental.

but if it’s a few-line modification I think that could improve the practical usability a lot.

Yes, considering we would not keep the incremental (one-time continuous) mode, the changes (wip) compared to the current PR would not be too much.
Allow me some further time to properly test this use-case and implementation.

Edit: this is now done and tested.

[ig11987]

I don't know about this script, it felt nice to have it during development to validate the integration, but I think it's too time sensitive and unreliable to put into CI.
But if it's not in CI it will get broken over time probably.