Tags: indexsupply/shovel
Tags
indexsupply.com/shove/docs: rewrite (#252) The new version of the docs has Overview and Reference sections. I was always frusterated that there wasn't a canonical description of the Config object. Now that exists as Reference: Config Object. But I felt that even with a Config Object Reference, there needs to be some high level description of the Config Object. So that exists as: Overview: Config. I also added a few other things.
shovel: bugfix. reorg may halt progress until restart The process for handling a reorg is as follows: 1. Shovel task gets latest remote block 2. Shovel task gets latest local block 3. Shovel decides which blocks to process 4. Shovel uses jrpc2 client (with cache) to download blocks 5. Shovel checks downloaded chain with latest local hash for reorg 6. If reorg then delete local (and all indexed data for that block) 7. GOTO 1 The problem is that step 4 may keep a block that has been removed from the chain in memory so that subsequent requests are given the removed block and therefore all new blocks are unable to be connected. This commit introduces the concept of maxreads for the block/header cache. When a task hits the maxreads for a particular block (segment) then the block will be removed from the cache. This approach maintains our ability to reduce downloaded data for concurrent tasks while providing a way for Shovel to make progress in the event of temporary, corrupt data. The maxreads is set in shovel's task setup to be equal to the total number of integrations configured. This should at least provide a smart way to reduce the time waiting for the invalid data to be pruned. Another approach would have been to have the task reset the cache when a reorg is detected, but that seemed too complicated. I don't like the idea of leaking the cache abstraction outside of the jrpc2 package. Another idea is to change the Get API to take a hash in addition to num/limit. But this makes the API more complicated to use.
shovel/web: bugfix. localhost dashboard access This code incorrectly assumed that if you were accessing the dashboard over localhost then you were running shovel in dev and would have access to the html templates on the fs. This is incorrect. You may want to access the dashboard over localhost using the released binaries, in which case you don't have the html on the fs. So now we always fall back to the html in the compiled binary. We also log in case something is going wrong in dev.
shovel: expose poll duration for eth source (#246) It is common for free node services to aggressively rate limit requests. This impacts both backfill and incremental indexing. For backfilling, it is advisable to use low values for batch_size and concurrency. For example: batch_size=10 and concurrency=1. For incremental indexing, it is now advisable to increase the poll_duration for a source. For example: poll_duration=1s Both of these knobs will reduce the amount of requests that are sent to a hosted node.
PreviousNext