Resolve the bottleneck created by the chain runtime in hermes

[ancazamfir]

Crate

relayer

Summary

I was debugging hermes recently on a testnet on a channel where lots of packets have been relayed. I noticed:

• when starting hermes the client worker does a lot of RPCs to retrieve headers (from src node and from dst Tx) and delays significantly the packet workers
  • partly addressed by Allow more granular configuration of Hermes modes of operation #1518
• after each relay operation, the client worker retrieves the header from the Tx on dest chain and then the header from the source chain, again delaying the packet worker thread
  • partly addressed by Misbehavior check triggered by UpdateClient event may run for a long time #1417
• in general all workers are affected by the runtime bottleneck, as all have to send their queries via the runtime channel and wait in line, possible after a series of heavy query requests.
  • this is a long-term architectural change, will probably be batched with other changes we do to the relayer architecture

Problem Definition

Proposal

In the long run we should allow queries to be done directly from the worker. There is no need to use the chain runtime for this. It should be only required for Tx.

In the short term segregate the client worker from the others, via configuration. This will allow a hermes instance to run in the background just to monitor for misbehaviour and keep clients fresh. Another instance could run for packet relaying and the slowdown caused by the client workers should be significantly less.

Fixing #1417 would also help.

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned