CachingChainHandle do not preserve ordering of chain queries

[soareschen]

Crate

ibc-relayer

Summary of Bug

The CachingChainHandle we introduced in #1932 do not preserve the ordering of request and response in case there is a cache hit.

In the original BaseChainHandle implementation, the requests are serialized through a single channel with requests coming in at a later time always get a response later than earlier requests. However with CachingChainHandle introduced, it would return from the method call immediately, even when earlier requests are queued up by the underlying ChainHandle. This may cause unexpected behavior, and may be the cause of why #1947 failed.

We may want to consider either to preserve the request/response ordering CachingChainHandle, or remove any faulty assumption that require the ChainHandle requests to be ordered. In the long term, we do want to remove that limitation so that multiple chain requests can be handled in parallel, in particular for the case of querying vs submitting transactions. So we would still need to investigate which part of the code assume the ChainHandle request/response are ordered.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned