Add retry mechanism for pov-recovery

[skunert]

While investigating flaky pov-recovery test behaviour, I found that it sometimes fails because data is still Unavailable on the relay chain.
Example Test
Node Logs

We are starting recovery as soon as we see a candidate pending availability. Since availability is not guaranteed, it can happen that the candidate is not yet available on the relay chain. We should have a backup mechanism that retries failed recovery requests.

[rphmeier]

It should maybe instead be that availability requests are queued until requisite bitfields are posted to the relay-chain. Although things might be available already, despite not being posted to the relay chain yet. Watching the offchain bitfield gossip would also be a way to learn about this faster.

[rphmeier]

Also, one thing I am curious about is at what point we actually initiate PoV recovery.

In principle, PoV recovery should only be necessary on collators in the case of collator block withholding attacks (when the collator sends the PoV to validators but doesn't share it with others). What I am getting at is that we shouldn't need PoV recovery unless there is either a malicious collator or the sync code isn't fetching the block properly for some reason.

In the issues we've seen in test networks, malicious collators are almost certainly not the underlying issue.

[skunert]

In principle, PoV recovery should only be necessary on collators in the case of collator block withholding attacks (when the collator sends the PoV to validators but doesn't share it with others). What I am getting at is that we shouldn't need PoV recovery unless there is either a malicious collator or the sync code isn't fetching the block properly for some reason.

Yes, this is the scenario. We use PoV recovery when a collator does not send its block announcement for whatever reason. We are currently sending block announcements for parachains when at least one validator has validated our PoV. If the submitting collator node dies in the meantime for whatever reason, the block is not announced. In that case, the other participants can recover.

The failing logs I posted in the opening description are from a zombienet test that provokes this scenario by disabling block announcements altogether.

Also, one thing I am curious about is at what point we actually initiate PoV recovery.

We trigger a recovery operation when we see a pending candidate in an incoming relay chain block and the candidate is unknown to our collator node (was not announced). In reality, the scenario should indeed be rare, the test just pushes it to the extreme.