Skip to content

v1.5

@ryandotsmith ryandotsmith tagged this 01 May 05:18
The process for handling a reorg is as follows:

1. Shovel task gets latest remote block
2. Shovel task gets latest local block
3. Shovel decides which blocks to process
4. Shovel uses jrpc2 client (with cache) to download blocks
5. Shovel checks downloaded chain with latest local hash for reorg
6. If reorg then delete local (and all indexed data for that block)
7. GOTO 1

The problem is that step 4 may keep a block that has been removed
from the chain in memory so that subsequent requests are
given the removed block and therefore all new blocks are unable
to be connected.

This commit introduces the concept of maxreads for the block/header
cache. When a task hits the maxreads for a particular block (segment)
then the block will be removed from the cache. This approach maintains
our ability to reduce downloaded data for concurrent tasks while
providing a way for Shovel to make progress in the event of temporary,
corrupt data. The maxreads is set in shovel's task setup to be equal to
the total number of integrations configured. This should at least
provide a smart way to reduce the time waiting for the invalid data to
be pruned.

Another approach would have been to have the task reset the cache when a
reorg is detected, but that seemed too complicated. I don't like the idea
of leaking the cache abstraction outside of the jrpc2 package.

Another idea is to change the Get API to take a hash in addition to
num/limit. But this makes the API more complicated to use.
Assets 2
Loading