[Core] Optimize reindex #1864
Closed
[Core] Optimize reindex #1864
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
random-zebra
force-pushed
the
202009_betterreindex
branch
from
46f3df3 to
788564a
Compare
|
Rebased. |
furszy
previously approved these changes
Sep 28, 2020
random-zebra
force-pushed
the
202009_betterreindex
branch
from
788564a to
2d30933
Compare
|
Rebased on master, after #1872 merge |
furszy
approved these changes
Oct 2, 2020
|
This is going to need additional work in order to function as intended. The intention of this optimization was to do an initial first pass through the blk files to build the chain index, then do a second pass activating said chain index. Given how our POS consensus works (validating the stake input), the first pass fails on the first POS block because, at this point, the blockIndex, For reference purposes, here is a table with full reindex timings for both mainnet and testnet using the current
|
||||||||||||||||||||||||||||||||||||||||
|
Good finding. Closing for now... this will need more work that can only be done after 5.0 |
random-zebra
added a commit
to random-zebra/PIVX
that referenced
this pull request
Oct 10, 2020
…oney supply dcc032c [Cleanup] Remove ZC_WrappedSerialsSupply no longer needed (random-zebra) 0e7921e [DB] Also prune invalid/fraudulent/frozen outputs from the coins view (random-zebra) b8647bd [Tests] Fix expected PIV supply in mining_pos_reorg (random-zebra) 40b7ea3 [DB] Prune zerocoin mints from coins DB at startup (random-zebra) b89bb91 [BUG] Don't try to spend zc mint coins from the cache in DisconnectBlock (random-zebra) cfc06ed [Core] Don't add zerocoin mints to the coins cache (random-zebra) e8a782c [Cleanup] Remove unused function GetInvalidUTXOValue (random-zebra) db7d8fd [Core] Remove nMoneySupply global and RecalculatePIVSupply function (random-zebra) b674fde [DB] Remove money supply databasing (random-zebra) 373ee90 [Cleanup] Remove --reindexmoneysupply startup flag (random-zebra) 02c9aaa [RPC] Return total available utxo value as moneysupply (random-zebra) cdf9bc1 [Core] Implement CCoinsViewCache::GetTotalAmount (random-zebra) Pull request description: Instead of updating and databasing independently `nMoneySupply`, just use the sum of all unspent tx outputs, returned by the coins cache. This has the following advantages: - Cleaner code: we don't need to add special cases for exceptional one-time events, such as wrapped serials, accumulator recalculation, etc. - More robust: with the old code, `nMoneySupply` is databased at every block and does not follow the chainstate flushing strategy, so it's very easy to get an inconsistent state (see PIVX-Project#1873). Even more so, considering the absence of any consistency check between the actual chain height and the height at which the supply was updated. - More accurate number: this returns only the __spendable__ supply, so it excludes all unspendable outputs (OP_RETURN), such as proposals / finalized budgets collaterals, that were originally counted in. Those outputs, in fact, are effectively "burnt" coins and should not be part of the total supply calculation. Disadvantage: - Calculating the supply from the coins DB is a bit slow, but the only place where this is done is in the RPC `getinfo`. This PR removes the global variable `nMoneySupply` and all related code (Closes PIVX-Project#1873) and gets rid of the startup flag `--reindexmoneysupply` no longer needed (PIVX-Project#1864 introduces `reindex-chainstate`).⚠️ This also fixes the following bug with zerocoin txes: Currently, on master, `gettxoutsetinfo` reports a total amount of about **811 million PIV**! This is due to improper handling of zerocoin-related "utxos". Basically mints are added to the coins cache (`AddCoin` only skips OP_RETURN coins, not mints) and *never* pruned/spent (as they are skipped inside `UpdateCoins`), except in `DisconnectBlock`. Here we completely remove zerocoin mint outputs from the coins DB. This makes sense, as originally they were not meant to be linked when spending, so should have been considered unspendable outputs, and not included in the first place. With the introduction of Public Spends, mint outputs are now referenced when spending, but we don't need to keep them in the cache either, as double spending is checked only via the zerocoin serial (and, as said, the coins aren't even spent in `UpdateCoins`). We also remove all those utxos flagged as invalid. Removing zerocoin mints from the unspent coins DB also gives a nice reduction to the utxoset size on disk (about 37%, from 428 Mb to 270 Mb). ACKs for top commit: Fuzzbawls: ACK dcc032c Tree-SHA512: da50dd81b9e3a56291812cf35158e1858f5a0c6ebdf1e5018d1e1d20b834dbdccdaa4cb0d82830c6f0a405c71428355615fd57f582065b992d42fa9a7313235e
furszy
added a commit
that referenced
this pull request
Jun 28, 2021
73d7288 [Doc] Mention reindex-chainstate in the release notes (random-zebra) c45c0ec Skip MoneySupply update during reindex/reindex-chainstate (random-zebra) efe193f Scan for better chains in ThreadImport (random-zebra) 0aceea3 Make ProcessNewBlock dbp const and update comment (Pieter Wuille) 8255e82 [Cleanup] Remove fVerifyingBlocks global flag (random-zebra) 30c7aed Add -reindex-chainstate that does not rebuild block index (random-zebra) 79d5cb2 Verify DB with original default check-level 3 (random-zebra) ed96859 Reduce default number of blocks to check at startup (Pieter Wuille) d4516fa Log/report in 10% steps during VerifyDB (Jonas Schnelli) cce2c9e doc: Mention dbcache increase in release notes (random-zebra) 11dc022 [GUI] Update default db cache size in the options model (random-zebra) 9b7f764 Bump `-dbcache` default to 300MiB (Wladimir J. van der Laan) Pull request description: This was attempted already in #1864 and #1907. Third time's a charm. Now the speed is acceptable and (unlike what was happening in #1907) the process can be interrupted. Further, as it is performed in ThreadImport, the GUI is open and functional during the chainstate reindex, instead of being seemingly stuck at the splash screen. This PR also includes a few improvements coming from bitcoin#10919, bitcoin#8273, bitcoin#8136 and bitcoin#8611. Here's a quick comparison of the data I had testing here (empty wallet with GUI, empty pivx.conf / default values). The reindexes have been performed with 0 network connections. <table> <tr> <th>Mainnet (height=2847011)</th><th>Master</th><th>this PR</th> </tr><tr> <td>online sync</td><td>8 hr 13 min</td><td>6 hr 42 min</td> </tr><tr> <td>-reindex (offline)</td><td>7 hr 14 min</td><td>6 hr 21 min</td> </tr><tr> <td>-reindex-chainstate (offline)</td><td>N/A</td><td>6 hr 13 min</td> </tr><tr> <td>startup (synced node)</td><td>100 sec</td><td>83 sec</td> </tr><tr> <td>RAM (synced node)</td><td>1.65 GiB</td><td>1.5 GiB</td> </tr><tr> <td>utxo_cache (synced node)</td><td>28.9MiB</td><td>39.5MiB</td> </tr> </table> ACKs for top commit: furszy: re-ACK 73d7288, tested again. Tree-SHA512: fb828d89692ccb6105eae5411b70a3c7a129d98d3101121cf604171de1b818a8c68b3771cc69fcf0a04642fe1327ee7ce360a72ecd7ef019b1614bd83c081de5
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Another attempt at backporting bitcoin#7917.
Previously in #1334, but put in stand-by few months ago due to some issue after reindex.
Tested both syncing from scratch and using
-reindexflags. It seems all good now.Reindex is still slower than resync for some reason, but at least not as much as in master (about 63% faster now).