Test packet relaying on ordered channels

[adizere]

Summary

In preparation for releasing ICA on the hub, we would like to do some testing that channel ordering works as expected in Hermes.

Proposal

• set up ordered channels between 2 chains
• provoke out-of-order packet relaying by either manually instrumenting the relayer code to drop the first event out of a batch, or by evicting transactions from thee node's mempool before they commit to the blockchain

Stretch goal:

• benchmark the scalability properties of Hermes by deploying it on a server similar to real-world operator use, and assess how many channels can Hermes relay gracefully on such a machine
  • the bottleneck will likely be the chain software, not Hermes: confirm that by isolating the networks on machines separate from the one where Hermes is running.

Acceptance Criteria

• capture a packet ordering corner-case in an automated test using the Rust framework
  • ensure that Hermes enforces ordering by looking at packet sequence numbers on an ordered channel

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[mzabaluev]

Testing this in a real setup appears to require a modified Gaia with the transfer module hacked to use ordered channels.

I guess we should go for a test case simulating out-or-order packet relaying in the test framework.