Skip to content

Conversation

@algorandskiy
Copy link
Contributor

@algorandskiy algorandskiy commented Oct 6, 2021

Summary

Historically only account updates tracker was committing into DB. The PR changes this:

  1. dbRound management and commitSyncer business moved to trackerRegistry.
  2. account updates is used as a driver for committing tasks scheduling - by given commttedRound and dbRound it figures out offset in detlas for saving to the disk. This is basically old au.committedUpTo method`.
  3. commitRound replaced by prepareCommit that returns commit and post commit callbacks
  4. trackerRegistry has own commitRound that calls trackers' prepareCommit and commit/post commit procs if any.
  5. account updates still have a cached dbRound to be used in Lookup methods.

Test Plan

It is refactoring, use existing tests

@codecov-commenter
Copy link

codecov-commenter commented Oct 8, 2021

Codecov Report

Merging #3014 (a23acb8) into master (a957519) will increase coverage by 0.01%.
The diff coverage is 68.05%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3014      +/-   ##
==========================================
+ Coverage   43.65%   43.66%   +0.01%     
==========================================
  Files         391      391              
  Lines       86792    86860      +68     
==========================================
+ Hits        37890    37930      +40     
- Misses      42873    42895      +22     
- Partials     6029     6035       +6     
Impacted Files Coverage Δ
ledger/trackerdb.go 38.35% <60.00%> (ø)
ledger/acctupdates.go 64.67% <60.12%> (-1.78%) ⬇️
ledger/tracker.go 76.19% <76.34%> (+7.44%) ⬆️
ledger/ledger.go 63.25% <78.57%> (+0.20%) ⬆️
ledger/accountdb.go 65.62% <100.00%> (-0.28%) ⬇️
ledger/bulletin.go 93.33% <100.00%> (+0.47%) ⬆️
ledger/catchpointwriter.go 69.48% <100.00%> (ø)
ledger/metrics.go 100.00% <100.00%> (ø)
ledger/notifier.go 88.23% <100.00%> (+0.73%) ⬆️
ledger/txtail.go 84.50% <100.00%> (+0.44%) ⬆️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a957519...a23acb8. Read the comment docs.

@algorandskiy algorandskiy force-pushed the pavel/ledger-sync branch 2 times, most recently from d294984 to 963e521 Compare October 8, 2021 19:53
@algorandskiy algorandskiy marked this pull request as ready for review October 8, 2021 19:59
@algorandskiy algorandskiy marked this pull request as draft October 8, 2021 20:00
@algorandskiy algorandskiy marked this pull request as ready for review October 9, 2021 01:15
@algorandskiy algorandskiy force-pushed the pavel/ledger-sync branch 2 times, most recently from d3d06ff to e21a220 Compare October 11, 2021 02:53
@algorandskiy algorandskiy force-pushed the pavel/ledger-sync branch 4 times, most recently from 8d3e969 to 1574db9 Compare October 12, 2021 16:55
@algorandskiy algorandskiy requested review from cce and shiqizng October 12, 2021 18:08
ledger/ledger.go Outdated
Comment on lines +407 to +414
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these two methods are used for testing only. I think it would be cleaner to contain these inside the trackerRegistry.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the primary reason of having these is account updates initialization:

				// flush the account data
				// TODO: figure out how to move it the upper level
				au.ledger.scheduleCommit(blk.Round())

				// wait for the writing to complete.
				au.ledger.waitAccountsWriting()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok leaving it here for now - but it's not the right place for that. It looks to me as a "hint" that it doesn't really belong here. This part would "roll-forward" the database to be at the correct round ( i.e. x -> latest-320 ).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, it is part of account updates initialization, I did not figure out a better way to accessing upper level from an tracker.

Copy link
Contributor

@tsachiherman tsachiherman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the changes here are going to the right direction, but I'd suggest the following:
in trackerRegistry.commitRound, before calling prepareCommit on all the children, create an object called roundCommitContext.
have the roundCommitContext contains the offset, dbRound, lookback, plus all the temporary staging data we need while committing the current round.
Then, on each of the ledgerTracker(s), implement the following methods:
prepareCommit(*roundCommitContext)
commitRound(ctx, tx, *roundCommitContext)
postCommitRound(ctx)

and call all of them ( in all the cases ).

This pattern would allow you to use the go-interface more, and would allow us to break up the trackers even further ( i.e. the shared data can be used across the trackers, so we can have the merle trie hashes be in the same tracker as the catchpoint file generation.. )

Copy link
Contributor

@tsachiherman tsachiherman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.

@tsachiherman tsachiherman merged commit 507eb53 into algorand:master Oct 14, 2021
cce pushed a commit to cce/go-algorand that referenced this pull request Oct 28, 2021
## Summary

Historically only `account updates` tracker was committing into DB. The PR changes this:
1. `dbRound` management and `commitSyncer` business moved to trackerRegistry.
2.  `account updates` is used as a driver for committing tasks scheduling - by given `commttedRound` and `dbRound` it figures out offset in detlas for saving to the disk. This is basically old `au.committedUpTo` method`.
3. `commitRound` replaced by `prepareCommit` that returns commit and post commit callbacks
4. trackerRegistry has own `commitRound` that calls  trackers' `prepareCommit` and commit/post commit procs if any.
5. `account updates` still have a cached dbRound to be used in Lookup methods.

## Test Plan

It is refactoring, use existing tests
@egieseke egieseke mentioned this pull request Nov 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants