runtime_api subsystem queues requests internally and likely does redundant work

[eskimor]

Use the queue Luke

Instead of imposing back pressure on the incoming channel. Conceptually it is rather pointless to internally queue requests unbounded, while receiving messages over a bounded channel. The subsystem at least files a warning when that internal buffer reaches a certain size, still backpressuring on the receiving channel would the very least help in making issues visible immediately.

We now have metrics for ToF of messages through a channel, we also monitor channel sizes and will soon have a metric for whenever a channel gets full (and thus imposes back pressure) - we would like this tooling to expose issues as well as possible, so for this reason alone we should fix that.

A similar issue exists for PVF execution, it also uses internal unbounded channels.

Be lazy

What we also noticed, that we will execute requests in parallel and have a cache. The issue here is, that we will be doing unnecessary work if identical requests arrive at roughly the same time: Because we will spawn each one of them, instead of only spawning one and then serving the others from the cache once that one succeeds.

What we should be doing, is keep track of the requests that are currently being executed and not spawn identical requests, while they are running, but instead just wait for the result hitting the cache and then serve all of them.

[drahnr]

The short term fix to get insight here, is to use the metered::* types internally as well. That would not solve the in principal issue, but at least shed some light.

[eskimor]

I think that is hardly less work than the proper fix. From looking at the code, this should be a pretty trivial change.

[eskimor]

@sandreim once we have results on the cache misses we have because of this, please add them here so we get an idea how important this is. Also do we have data how much faster the cache is versus the execution? I am guessing the difference will be quite significantly, but do we know?

[sandreim]

Burned in paritytech/polkadot#5541 in Versi and data shows that there are indeed cache misses due to parallelizing same requests. ToF went up by 2.5x so and we need to move back to 4 parallel requests to see how much requests take on average to execute. With this data we can decide if it is worthy to prioritize the cache optimization sooner rather than later.

[sandreim]

The runtime calls are on average very cheap as it can be seen in the picture below. I think we should not implement this cache improvement now, but keep it for later, when we are out of big opportunities for optimisation.

[sandreim]

Created #822 for the cache improvement.