enhancement: store BlockData in the DB and ensure the indexers don't miss blocks #1297
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lostman
force-pushed
the
maciej/blockstore
branch
from
28d50ff
to
edafc3a
Compare
lostman
commented
Aug 25, 2023
| @@ -681,7 +711,7 @@ pub async fn last_block_height_for_indexer( | |||
| Ok(row | |||
| .try_get::<i32, usize>(0) | |||
| .map(|id| id.to_u32().expect("Bad block height.")) | |||
| .unwrap_or_else(|_e| 1)) | |||
| .unwrap_or(0)) | |||
There was a problem hiding this comment.
I think we need to return zero here, for consistency.
If an indexer processed blocks 1, 2, and 3, the result would be 3.
If an indexer processed block 1, the result would be 1.
If an indexer processed no blocks, we shouldn't return 1 (as we currently do) but 0.
I made changes to get_start_block to account for this.
lostman
commented
Aug 25, 2023
| @@ -387,7 +387,8 @@ pub async fn get_start_block( | |||
| .await?; | |||
| let start = manifest.start_block().unwrap_or(last); | |||
| let block = if *resumable { | |||
| std::cmp::max(start, last) | |||
| // if the last processed block is N, we want to resume from N+1 | |||
| std::cmp::max(start, last + 1) | |||
There was a problem hiding this comment.
If the indexer processed no blocks and last_block_height_for_indexer returned 0, we bump it to 1—the first new block it needs to process.
Similarly, if the indexer processed blocks 1, 2, 3, and last_block_height_for_indexer returned 3, we bump it to 4—the first new block.
lostman
force-pushed
the
maciej/blockstore
branch
from
806525c
to
912d9e2
Compare
lostman
force-pushed
the
maciej/blockstore
branch
from
2a0c8ee
to
52cfe6c
Compare
lostman
force-pushed
the
maciej/blockstore
branch
from
14096e0
to
badf8c3
Compare
lostman
force-pushed
the
maciej/blockstore
branch
from
ca1c432
to
bd0c68a
Compare
|
@lostman Is this PR good to close? |
|
Deprecated in favor of #1369 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds
index_block_datatable in which we store serializedBlockDatastructs. We can use these to get a new indexer up to speed without fetching blocks from the client again.The table contains a trigger that ensures we have a full range of blocks—no
block_heightmay be missing.Additionally, this PR adds the same trigger to each indexer's
indexmetadataentitytable. Again, this ensures that an indexer must process each block in sequence. If the indexer fails to process a block or processes blocks out of order, the trigger will raise an exception (and the indexer's transaction will fail).Note
The trigger ensures the indexer can't process blocks out of order or miss any blocks. However, it doesn't stop the indexer from running—it simply doesn't progress. To stop the indexer, we need the WASM error codes #1337 and return an error (and trigger an early exit!) when
put_objectfails.Testing steps
Please provide the exact testing steps for the reviewer(s) if this PR requires testing.
Changelog
Please add neat Changelog info here, according to our Contributor's Guide.