indexer: exact epoch tx count #12395
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
gegaowp
requested review from
666lcz,
wlmyng,
patrickkuo,
bmwill,
longbowlu and
healthydeve
and removed request for
666lcz and
wlmyng
| @@ -235,6 +235,10 @@ pub trait IndexerStore { | |||
| ) -> Result<(), IndexerError>; | |||
|
|
|||
| async fn persist_epoch(&self, data: &TemporaryEpochStore) -> Result<(), IndexerError>; | |||
| async fn count_network_transaction_previous_epoch( | |||
There was a problem hiding this comment.
I think a function that counts transactions in epoch X makes more sense than in epoch X-1
There was a problem hiding this comment.
this is to fetch the last network_total_transactions value of previous epoch in checkpoints table, we cannot count the tx of current epoch yet, b/c the checkpoint is likely not committed yet.
the more I think, the more I want to move all epoch related calculation & DB commits to an async processor, such that this async processor will just wait until all checkpoints have been complete on the previous epoch and then start processing.
There was a problem hiding this comment.
I was suggesting:
fn count_network_transaction_in_epoch
and then call count_network_transaction_in_epoch(n-1)
will leave it up to you though
There was a problem hiding this comment.
yes, I saw what you meant, I wanted to calculate the total tx of current epoch based on
- committed row of last checkpoint of previous epoch, where the network_total_tx is tx count since genesis;
- the current checkpoint row in mem, which is the last row of the current epoch and may or may not committed
such that the difference is the tx count of current tx.
b/c the current checkpoint row might not have been committed, thus running query of count_network_transaction_in_epoch could generate an inaccurate result.
If we do that with an async processor, it would be this issue b/c we can always have triggering condition to make sure the all checkpoints have been committed before processing the epoch
gegaowp
force-pushed
the
indexer-epoch-tx-count
branch
from
63ecaa4 to
fd6bbcf
Compare
gegaowp
force-pushed
the
indexer-epoch-tx-count
branch
from
fd6bbcf to
70800c4
Compare
## Description
Before this PR, tx count was updated within an epoch. This is
problematic on backfill when data racing happens, as a result, epoch tx
count is not accurate.
Also removed the epoch updating on each checkpoint b/c it has caused
constant DB commit errors like
```
2023-06-07T23:13:27.759942Z INFO sui_indexer::store::pg_indexer_store: Persisting epoch 19
2023-06-07T23:13:36.865246Z WARN sui_indexer::handlers::checkpoint_handler: Indexer epoch commit failed with error: PostgresWriteError("could not serialize access due to read/write dependencies among transactions"), retrying after 100 milli-secs...
2023-06-07T23:13:36.965945Z INFO sui_indexer::store::pg_indexer_store: Persisting epoch 19
2023-06-07T23:13:46.059553Z WARN sui_indexer::handlers::checkpoint_handler: Indexer epoch commit failed with error: PostgresWriteError("could not serialize access due to read/write dependencies among transactions"), retrying after 100 milli-secs...
2023-06-07T23:13:46.160704Z INFO sui_indexer::store::pg_indexer_store: Persisting epoch 19
```
## Test Plan
Build image and test with mainnet, verify in the DB.
---
If your changes are not user-facing and not a breaking change, you can
skip the following section. Otherwise, please indicate what changed, and
then add to the Release Notes section as highlighted during the release
process.
### Type of Change (Check all that apply)
- [ ] protocol change
- [x] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration
### Release notes
Description
Before this PR, tx count was updated within an epoch. This is problematic on backfill when data racing happens, as a result, epoch tx count is not accurate.
Also removed the epoch updating on each checkpoint b/c it has caused constant DB commit errors like
Test Plan
Build image and test with mainnet, verify in the DB.
If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.
Type of Change (Check all that apply)
Release notes