Fix performance of StateCompute RPC on Archival nodes

[aarshkshah1992]

StateCompute for an epoch near the head can take 1 minute to complete on an archival node. For epochs far in the past, it can 4-5 minutes. This is only for a single request and performance degrades further for multiple requests. This makes it extremely hard to build a Filecoin explorer(which needs archived state) as it can take months to index the entire chain.

We should get access to an archival node and profile the StateCompute RPC API to find performance hotspots and fix them.

(This was learned when talking with Zondax on 2024-07-24.)

[dubuqingfeng]

Is there a solution for this?

[rvagg]

Not yet, we're not entirely sure what the issue might be other than badger being really bad at large amounts of data and probably really bad at data that's not touched very much (historical).