Batch eth_getStorageAt

[FeurJak]

Is there an available method to do a batch eth_getStorageAt to retrieve multiple storage variables? I know the exact indexes all-ready, but it's inefficient for me to make a single rpc call per variable.

[FeurJak]

Okay, just discovered eth_getProof.... but this seems to take quite a bit of time just to get 3 variables (~1.2ms)..

[fjl]

You can also use a JSON-RPC batch request. While this may still incur some internal overhead for each storage slot, it will at least save the overhead of sending the requests.

[FeurJak]

You can also use a JSON-RPC batch request. While this may still incur some internal overhead for each storage slot, it will at least save the overhead of sending the requests.

I'm using go & I"m calling "eth_getStorageAt" via an rpclient (i..e. rpclient.CallContext). I can write my own batch request but I believe these requests are still handled one by one internally.

My understanding is that a new stateDB and stateObj has to be created for every request, so I'm looking to create a new function (i.e. getBatchStorageAt, GetBatchState, GetBatchCommitedState...) which will return an array of storage variables based on an array of indexes for 1 address.

[fjl]

I understand you are talking about creating a new method here, just trying to avoid it. Let's explore the existing alternatives before adding a new RPC method.

If the performance of RPC batch is not enough for you, you can also get multiple slots using eth_call. In geth, this method accepts a third parameter containing 'state overrides' (see documentation). To get any number of storage slots, you could encode your request as an EVM program and run it in the context of your target contract.

[fjl]

My understanding is that a new stateDB and stateObj has to be created for every request

This step should be kind of cheap because a lot of things are cached internally. I'd prefer you try the RPC batch first and check if the performance is good enough for your use case.

[FeurJak]

My understanding is that a new stateDB and stateObj has to be created for every request

This step should be kind of cheap because a lot of things are cached internally. I'd prefer you try the RPC batch first and check if the performance is good enough for your use case.

Fair point, I'll measure the performance of all-ready available methods before going forward with creating new methods.

With Eth_Call, not too clear about the stateDiff format as it's not given in the examples, would you know what the format for that parameter is? (i.e. "stateDiff": {"0x0000000000000000000000000000000000000001": {"code": "0x6080604052600760005260206000f3fe"}} ) ?

[georgercarder]

@FeurJak , could your batching needs be met with their GraphQL api?

[s1na]

This is the best graphql query I could come up with for re this issue:

query slots($blocknum: Long!, $addr: Address!, $key0: Bytes32!, $key1: Bytes32!, $key2: Bytes32!) {
  slot0: block (number: $blocknum) {
    account(address: $addr) { storage (slot: $key0)}
  },
    slot1: block (number: $blocknum) {
    account(address: $addr) { storage (slot: $key1)}
  },
    slot2: block (number: $blocknum) {
    account(address: $addr) { storage (slot: $key2)}
  }
}

With the variables being something like (example is a random contract from Goerli):

{
  "blocknum": "6004069",
  "addr":"0xD3909f56dFb3B6F11Fc0d82Ef6022C6485EB0AE0",
  "key0":"0x0000000000000000000000000000000000000000000000000000000000000003",
  "key1": "0x7461cd233bf72e6c1fab8f2ffafce9cd48c96e1b422444c78aeae2cdd60843c8",
  "key2": "0x7461cd233bf72e6c1fab8f2ffafce9cd48c96e1b422444c78aeae2cdd60843c9"
}

This could've been re-written to become a bit shorter had graphql supported parameterized fragments (graphql/graphql-spec#204). Right now this is what we're stuck with which sucks a bit if you have a dynamic and variable list of slots you want to query each time. You'll have to do some string juggling to construct the query.

I'm going to try that next and compare the performance with a doing several eth_getStorageAt as a next step.

[s1na]

Surprisingly for large numbers of storage slots requested GraphQL can be slower than sending a series of separate JSON-RPC requests. I wrote a script which gets the prestate for a block, turns that into an access-list and then does first: a series of eth_getStorageAt calls, and then fills a query such as the one above with all the storage slots. Here are the results for a few mainnet blocks (on a live node, so data noisy). Note the slow graphql response when number of storage slots exceeds 1000:

$ node index.js 14121860
Access-list has 215 accounts with 1430 storage slots
json-rpc requests took 571ms
graphql request took 1214ms

$ node index.js 14121859
Access-list has 48 accounts with 358 storage slots
json-rpc requests took 188ms
graphql request took 73ms

$ node index.js 14121858
Access-list has 92 accounts with 680 storage slots
json-rpc requests took 310ms
graphql request took 233ms

$ node index.js 14121857
Access-list has 219 accounts with 1718 storage slots
json-rpc requests took 749ms
graphql request took 4901ms

$ node index.js 14121856
Access-list has 152 accounts with 1544 storage slots
json-rpc requests took 622ms
graphql request took 1134ms

$ node index.js 14121855
Access-list has 13 accounts with 109 storage slots
json-rpc requests took 72ms
graphql request took 15ms

[FeurJak]

Sorry for not replying to you guys, had no idea that this thread was still alive.

I unfortunately can't go down the GraphQL route for my application.
I ended up with writing my own eth_GetBatchStorageAt on /internal/ethapi/api.go & state.GetStatesFromStorage(addresses,keys) , core/state/statedb.go:

func (s *PublicBlockChainAPI) GetBatchStorageAt(ctx context.Context, addresses []string, keys []string, blockNrOrHash rpc.BlockNumberOrHash) ([]common.Hash, error) { state, _, err := s.b.StateAndHeaderByNumberOrHash(ctx, blockNrOrHash) if state == nil || err != nil { return nil, err } res := state.GetStatesFromStorage(addresses,keys) return res[:], state.Error() }

func (s *StateDB) GetStatesFromStorage(addrs []string, hashes []string) []common.Hash { var _res []common.Hash for _, _addr := range(addrs) { stateObject := s.getStateObject(common.HexToAddress(_addr)) if stateObject != nil { _res = append(_res, stateObject.GetStatesFromStorage(s.db, hashes)...) } else { for i := 0; i < len(hashes); i++ { _res = append(_res, common.HexToHash("")) } } } return _res }

Current solution is very quick.

[s1na]

@FeurJak I'm trying to find the fastest way to get a batch of storage values without having to add a new method to the API and I kinda hijacked your thread to share the results. Sorry about that :) Feel free to mute the thread if you're not interested in this anymore.

By splitting the storage slots and doing multiple GraphQL queries (say each with 100 slots) I managed to gain a 3-5x speedup compared to JSON-RPC. I updated the gist with the new script if anyone's interested. 3-5x is still not the speed-up I was hoping for so I'll keep the thread open for now in case I find another optimization.

[FeurJak]

@s1na I'm very much interested ! I think the batch storage lookup requirement would be needed by many others and the GraphQL looks to be a viable solution without adding a new method to the API.

[FeurJak]

@s1na I am curious as to why it would provide a 3-5x speedup compared to JSON-RPC though.

[fjl]

@s1na Did you find out where the slowdown in big queries comes from? I find it surprising that this splitting of queries is necessary.