Add pagination support for clear packets

[adizere]

Problem

In production, Hermes often encounters the problem of needing to clear many (hundreds) of packets on a given path. This can take a very long time, a problem compounded due to full nodes being slow at times. If a node is slow and Hermes takes a long time (on the order of minutes) to build a transactions for relaying some packets, that transaction might prove to be invalid or outdated by the Hermes finally submits it, making the whole process very painstaking.

The problem is that Hermes does bulk actions while trying to clear the packets on a path. This can be seen in the following spots that are on the critical path of clear packets CLI, for example:

• https://github.com/informalsystems/ibc-rs/blob/edc9a2cb50ed774ade03ccb846661ea49caf5b38/relayer/src/chain/counterparty.rs#L310-L314
  • this method fetches all pages of commitments on a chain
• https://github.com/informalsystems/ibc-rs/blob/edc9a2cb50ed774ade03ccb846661ea49caf5b38/relayer/src/chain/counterparty.rs#L345-L348
  • given all pages of commitments on a chain, this methods fetches all the unreceived packets for those commitments
• https://github.com/informalsystems/ibc-rs/blob/edc9a2cb50ed774ade03ccb846661ea49caf5b38/relayer/src/chain/counterparty.rs#L379-L384
  • Hermes fetches all pages of acks

Acceptance criteria

Discussion

Hermes CLI for packet clearing should provide support for splitting the workload in multiple, smaller batches, to make this command more tractable. This can be done, for instance, with any of the following suggestions:

A. By adding a flag clear packets ... --limit X to specify that Hermes should limit itself to fetching only X amount of packets/acks (instead of all pages). The limit option should probably be mandatory.

B. Alternatively, we could handle splitting into smaller batches of X messages automatically within the implementation of clear packets and have a simpler interface.

The advantage of A. is that the command is a lot more interactive and will finish faster. The problem is that the operators will not know if there will still be packets left to relay, so they will need to repeatedly invoke clear packets --limit X until this method returns an empty result (signalling there was nothing to clear).

The advantage of B. is that this command is smarter and simpler. The disadvantage is that will take longer to finish. On a high-activity path with constant activity, depending on the implementation and how we split-up the batches, I can even imagine that this method will constantly run for as long as there are packets being created, which is a problem (the CLI should return, instead of grabbing & relaying every new packet).

Other solutions might exist. We should evaluate the pros and cons through the lenses of operator user experience and simplicity.

Tasks

• add integration tests to assert the correctness of new behavior

[romac]

Ideally the command would use pagination and display its progress as it clears the packets, perhaps with a nice TUI progress bar?

On a high-activity path with constant activity, depending on the implementation and how we split-up the batches, I can even imagine that this method will constantly run for as long as there are packets being created

That's indeed an issue, not clear how to avoid that. Perhaps by not trying to clear packets at a higher height than the one the chain was at when the command was started? ie. the command would clear packets from oldest to newest and stop once it sees a packet at a height higher than the starting height

[romac]

Alternatively, query packet pending could return all the information needed to clear the packets it gathered, and this info could be piped into clear packets via stdin so that the command would only clear those packets. Seems a bit more involved and not a Cli pattern we've really explored so far.

[adizere]

Closing in favor of #2087