Tie the `payable_scanner` with the `pending_payable` scanner and prefigure the retry semicycles

[bertllll]

In the presence of the current state of things, it seems undesirable having those two scanners pending_payable_scanner and payable_scanner loosely configurable. It can cause an adverse effect if the person doesn't know what they are doing, and, it's been of a small advantage if otherwise, honestly. It comes to be more and more obvious that we can keep them distanced in a fixed manner.

The keeping the configuration of those other scan intervals enabled can be also questioned but it doesn't belong into this card.

We can actually run these two scanners consecutively, with basically no interval considered at all. We should definitely make sure the pending_payable_scanner goes always first! That's because it will be this scanner which will look after the status column. If it resolves a transaction attempt a failure (statuses beginning with f, followed by an opcode), it will be later picked up by the payable_scanner and included in the upcoming payment attempt, then executed immediately.

Update: Cycles and semicycles

Pursuing the best design for the moment and trying to implement the retry mechanism more effectively, we started to question the original idea of waiting for the check simply until the upcoming try whose timing is set by the payable interval. Instead we've come up with a system of dynamic semicycles and from there derived cycles.

In its utmost simplicity, it abides by this pattern:

P
Genuine scan for new payables producing payments
|
"retry interval"
Possibly also configurable
|
PP/P
This is where most of the logic is centered. This scan will always run at least once within one "payable interval", which basically becomes the distance between two genuine payable scans, without any regards to pending payments. In fact, by the time the payable scanner seeks for new payments, there will never be any pending payments anymore. Here is how we will enforce it. When this bundled scanner is ending it will always evaluate if any tracked payments were found still pending. Until there are some, we will always command another retry. That implies that the "regular" payable scan can be theoretically postponed because we do not want it to collide with the verification scanner. If necessary, the repeated cycles of the verification scanner can roll through the regular milestone and will not stop until all known pending payments are taken care of.
That is the ultimate condition for this verification scanner to conclude that the standard scanner should be scheduled.
Now, there are two positions in which this may happen, and it determines the timing of the newly scheduled payable scanner.
If the retries have so far taken less than a single payable interval, then we will compute how much time remains until the end of it, this will then be used as the time advancement of the scanner being scheduled.
On the other hand, if the row of retries have already gone past the length of the large interval, we will simply schedule it just immediately, "wasting no more time".
|
P
|

The elegance of this design is centered in the assurance that every next scanner will always begin after the earlier just ended. Nether the PP/P scan nor just the P scan will start off during another scan is yet in progress. Not doing this, it's true that we'd probably intuitively just prevent the new attempt and we'd end up with just one a time like here. The difference for the relationship of PP/P closely followed by P is that, going the other depicted direction, we'd have to cancel this second, and just starting, scan, but how would we make this loss up so that we do as much as possible to run the payable scanner in decent repetitions. We'd run into the behavioral limits of the actor model there, having to compensate it in some complicated way.

Thoughts on the implementation:

Visit those places where we refer to three different scan intervals. Introduce two big intervals and one small for the retries. It is likely that we'll end up with terms similar to "payable scanner interval", "payment_retry_interval" and "receivable_scanner_interval".

!! The retry interval should probably be defaulted for each chain specifically (it is strongly influenced by the design of the blockchain and the activity in the surrounding network).

• the scan config for Accountant which will imply a change in the piece of code that gives the initial stimulus to the scans cycle to get going; remember, newly we want to call the two scanners always together and therefore it would be nice to write a method containing just these two calls and we should always call this method anytime we want to run the scan (the aftereffect of this entire change will also be that we'll get shorter of one Actor message which was meant for the PendingPayableScanner, handed to it by the Accountant which used to send it to itself in the given interval again and again)
• the code of the Bootstrapper which parses the inputs of the program and provides the interval in a structured form to become part of the BootstrapperConfig which eventually nurture all the needs of the starting Actors (the collected data is used in the process of building and starting them); this will be most prevalent in the file combined_params hosting these structures and allowing the parsing; the scan intervals trio is treated in there too and therefore needs to be changed,
• this is hard to tell from just top of my head but maybe a change in the clap schema or at least in the help message for scan-intervals will be required too
• change in the scans command is also required, removing the option for scans pending-payable; that implies a little rework in the masq CLI, plus in the docs covering the websocket messaging so that the front-end team hypothetically also knows - even though I doubt they plan to implement this command in their UI.

Note: Ensure that we give some smart boundaries to the configurable intervals. For example, we must ensure that the retry mechanism isn't bigger than half the payable interval or something of this reasoning

[kauri-hero]

Discussed adding to this card, an extra piece of database migration for the --scan-intervals configuration parameter so it ignores any presence of value for pending-payables scan interval, since that will be incorporated into the overall payables-scan function.
Also be mindful that the --scan UI command also has a command for executing a pending-payables scan, so review that to ensure it does not conflict with the changes to pending-payables scanning
@bertllll @utkarshg6