Skip to content

Tags: indexsupply/shovel

Tags

v1.6

Toggle v1.6's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
indexsupply.com/shove/docs: rewrite (#252)

The new version of the docs has Overview and Reference sections. I was
always frusterated that there wasn't a canonical description of the
Config object. Now that exists as Reference: Config Object. But I felt
that even with a Config Object Reference, there needs to be some high
level description of the Config Object. So that exists as:
Overview: Config.

I also added a few other things.

v1.5

Toggle v1.5's commit message
shovel: bugfix. reorg may halt progress until restart

The process for handling a reorg is as follows:

1. Shovel task gets latest remote block
2. Shovel task gets latest local block
3. Shovel decides which blocks to process
4. Shovel uses jrpc2 client (with cache) to download blocks
5. Shovel checks downloaded chain with latest local hash for reorg
6. If reorg then delete local (and all indexed data for that block)
7. GOTO 1

The problem is that step 4 may keep a block that has been removed
from the chain in memory so that subsequent requests are
given the removed block and therefore all new blocks are unable
to be connected.

This commit introduces the concept of maxreads for the block/header
cache. When a task hits the maxreads for a particular block (segment)
then the block will be removed from the cache. This approach maintains
our ability to reduce downloaded data for concurrent tasks while
providing a way for Shovel to make progress in the event of temporary,
corrupt data. The maxreads is set in shovel's task setup to be equal to
the total number of integrations configured. This should at least
provide a smart way to reduce the time waiting for the invalid data to
be pruned.

Another approach would have been to have the task reset the cache when a
reorg is detected, but that seemed too complicated. I don't like the idea
of leaking the cache abstraction outside of the jrpc2 package.

Another idea is to change the Get API to take a hash in addition to
num/limit. But this makes the API more complicated to use.

v1.4

Toggle v1.4's commit message
indexsupply.com/shovel/docs: describe how to use traces

v1.3

Toggle v1.3's commit message

Verified

This commit was signed with the committer’s verified signature.
ryandotsmith Ryan Smith
shovel: add prometheus metrics endpoint (#248)

related: #238

v1.2

Toggle v1.2's commit message

Verified

This commit was signed with the committer’s verified signature.
ryandotsmith Ryan Smith
wpg: create index can include order term

v1.1

Toggle v1.1's commit message

Verified

This commit was signed with the committer’s verified signature.
ryandotsmith Ryan Smith
shovel/web: bugfix. localhost dashboard access

This code incorrectly assumed that if you were accessing the dashboard
over localhost then you were running shovel in dev and would have access
to the html templates on the fs. This is incorrect. You may want to
access the dashboard over localhost using the released binaries, in
which case you don't have the html on the fs. So now we always fall back
to the html in the compiled binary. We also log in case something is
going wrong in dev.

v1.0

Toggle v1.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
shovel: expose poll duration for eth source (#246)

It is common for free node services to aggressively rate limit requests.
This impacts both backfill and incremental indexing.

For backfilling, it is advisable to use low values for batch_size and
concurrency. For example: batch_size=10 and concurrency=1.

For incremental indexing, it is now advisable to increase the
poll_duration for a source. For example: poll_duration=1s

Both of these knobs will reduce the amount of requests that are sent to
a hosted node.

v1.0beta

Toggle v1.0beta's commit message

Verified

This commit was signed with the committer’s verified signature.
ryandotsmith Ryan Smith
shove: ui tweak. update demo config

v0.1.8

Toggle v0.1.8's commit message
abi: bugfix. Decode returned incorrect n value for dynamic types

If the length was already a multiple of 32 then the previous calculation
added 32 incorrectly.

v0.1.7

Toggle v0.1.7's commit message
genabi: bugfix. handle case where log has no topics