Load test Requests per second

[ValarDragon]

We need to find out how many requests per second a node can handle.

For now, we should probably make such a benchmark with

• Getting all pools
• Getting total liquidity
• Calculating Rewards
• spot price of a few pools

We can get a better profile of requests to run from @dogemos later.

There are some mutex locks in different places in the stack (e.g. tendermint/tendermint#6048, IAVL) which we should see if we can either fix, or skirt the need to use.

[tac0turtle]

It seems like many of the requests you mention above are sdk related. I would love to test if we get any performance improvements by removing most of this code cosmos/cosmos-sdk#8549 in favor of setting a RWmutex in the sdk opposed to handing it off to tendermint.

This will for sure fix tendermint related query issues

[ValarDragon]

100% agreed, this was mainly motivated because of the lack of concurrency.

However, I'm not at all convinced that the SDK isn't going to have weird locks elsewhere, e.g. every telemetry call making a mutex contention, even if telemetry isn't used on that ABCI connection!