Conversation
kompotkot
reviewed
May 29, 2023
|
|
||
| # Historical crawler status config | ||
|
|
||
| HISTORICAL_CRAWLER_STATUSES = { |
Contributor
There was a problem hiding this comment.
I suggest to use Enums, but let's try dictionary)
Comment on lines
233
to
235
| if block is None: | ||
| logger.warning( | ||
| f"Failed to find deployment block for {address}, code: {code}" |
Contributor
There was a problem hiding this comment.
I suggest to throw exception here
added 3 commits
May 29, 2023 16:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add atuation about historical crawl of each new subscription.
Depending on Spire client update
Tags tasks schema. https://drive.google.com/file/d/1z_jsWnqtxKSU-kHD6RNnaRdMoR9_PVRU/view?usp=sharing
General logic
For each blockchain, create two worker instances separate for events and transactions. These workers will read a
'moonworm-jobs'journal.Workers will collect abi's data for newly added subscriptions to addresses and generate a list of tasks in the form of events or functions to be crawled. The entries should have the tags
['moonworm_task_pickedup:True','historical_crawl_status:pending'].The worker processes will pick up these jobs and update their status to
'historical_crawl_status:in_progress'.For every iteration over a batch of blocks, the worker process will update the status of each entry to indicate the progress of the crawl. The progress is represented as
status:x.yy, wherex.yyis the percentage of the crawl that has been completed for a specific address.The progress percentage is calculated using the formula:
latest_block- get just in start and it current block on node.current_block- block between latest_block and deployment_blockThis gives the percentage completion of the crawl for the given address.
Once the progress percentage is 100 or more, the worker process will set the entry status to
'historical_crawl_status:finished'indicating that the crawl job for that particular address has been completed.Run command:
new subcommands:
--only-functions- only method calls.--tasks-journal- Get all tasks from moonworm journal.--find-deployed-blocks- use moonworm find deployment functionality.How to test these changes?
Related issues