v1.5
ryandotsmith
tagged this
01 May 05:18
The process for handling a reorg is as follows: 1. Shovel task gets latest remote block 2. Shovel task gets latest local block 3. Shovel decides which blocks to process 4. Shovel uses jrpc2 client (with cache) to download blocks 5. Shovel checks downloaded chain with latest local hash for reorg 6. If reorg then delete local (and all indexed data for that block) 7. GOTO 1 The problem is that step 4 may keep a block that has been removed from the chain in memory so that subsequent requests are given the removed block and therefore all new blocks are unable to be connected. This commit introduces the concept of maxreads for the block/header cache. When a task hits the maxreads for a particular block (segment) then the block will be removed from the cache. This approach maintains our ability to reduce downloaded data for concurrent tasks while providing a way for Shovel to make progress in the event of temporary, corrupt data. The maxreads is set in shovel's task setup to be equal to the total number of integrations configured. This should at least provide a smart way to reduce the time waiting for the invalid data to be pruned. Another approach would have been to have the task reset the cache when a reorg is detected, but that seemed too complicated. I don't like the idea of leaking the cache abstraction outside of the jrpc2 package. Another idea is to change the Get API to take a hash in addition to num/limit. But this makes the API more complicated to use.