Commits on Oct 17, 2024

1. rpc v2: backpressure chainHead_v1_storage (#5741) …

   Close #5589
   
   This PR makes it possible for `rpc_v2::Storage::query_iter_paginated` to
   be "backpressured" which is achieved by having a channel where the
   result is sent back and when this channel is "full" we pause the
   iteration.
   
   The chainHead_follow has an internal channel which doesn't represent the
   actual connection and that is set to a very small number (16). Recall
   that the JSON-RPC server has a dedicate buffer for each connection by
   default of 64.
   
   - Because `archive_storage` also depends on
   `rpc_v2::Storage::query_iter_paginated` I had to tweak the method to
   support limits as well. The reason is that archive_storage won't get
   backpressured properly because it's not an subscription. (it would much
   easier if it would be a subscription in rpc v2 spec because nothing
   against querying huge amount storage keys)
   - `query_iter_paginated` doesn't necessarily return the storage "in
   order" such as
   - `query_iter_paginated(vec![("key1", hash), ("key2", value)], ...)`
   could return them in arbitrary order because it's wrapped in
   FuturesUnordered but I could change that if we want to process it
   inorder (it's slower)
   - there is technically no limit on the number of storage queries in each
   `chainHead_v1_storage call` rather than the rpc max message limit which
   10MB and only allowed to max 16 calls `chainHead_v1_x` concurrently
   (this should be fine)
   
   - Iterate over 10 accounts on westend-dev -> ~2-3x faster
   - Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster
   - Fetch 1024 descendant values -> ~500x faster
   
   The reason for this is because as Josep explained in the issue is that
   one is only allowed query five storage items per call and clients has
   make lots of calls to drive it forward..
   
   ---------
   
   Co-authored-by: command-bot <>
   Co-authored-by: James Wilson <james@jsdw.me>

   

   niklasad1 and jsdw committed Oct 17, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 32f93a7
   • Browse repository at this point

   Copy the full SHA

   32f93a7 View commit details

   Browse the repository at this point in the history