core: separated databases for block data #2227
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fynnss
reviewed
Feb 22, 2024
flywukong
reviewed
Feb 26, 2024
flywukong
reviewed
Feb 26, 2024
jingjunLi
force-pushed
the
impl_block_store
branch
from
a8efd8e to
3c82f4b
Compare
flywukong
reviewed
Feb 28, 2024
node/node.go
Outdated
|
|
||
| if config.SeparateDB { | ||
| log.Info("SeparateDB is enabled, open block database, %s", config.DatabaseBlock) | ||
| blockStore, err = n.OpenBlockDatabase(name, config.DatabaseHandles-chainDataHandles, config.DatabaseBlock, namespace, readonly) |
There was a problem hiding this comment.
config.DatabaseHandles-chainDataHandles euqals zero , it it no meaning.
You can consider allocating 1/10 of the handlers to the block store
There was a problem hiding this comment.
Cache and handles usage is modified as follows:
There was a problem hiding this comment.
We have a new Resource allocation rules:
- Allocate a fixed percentage of memory for chainDb based on chainDbMemoryPercentage & chainDbHandlesPercentage.
- Allocate a fixed size for blockDb based on blockDbCacheSize & blockDbHandlesSize.
- Allocate the remaining resources to stateDb.
fynnss
reviewed
Feb 28, 2024
fynnss
reviewed
Feb 28, 2024
jingjunLi
changed the title
fynnss
reviewed
Mar 11, 2024
fynnss
reviewed
Mar 11, 2024
flywukong
reviewed
Mar 12, 2024
flywukong
reviewed
Mar 12, 2024
flywukong
reviewed
Mar 12, 2024
flywukong
reviewed
Mar 12, 2024
flywukong
reviewed
Mar 12, 2024
flywukong
reviewed
Mar 12, 2024
node/node.go
Outdated
| @@ -73,10 +74,13 @@ const ( | |||
| initializingState = iota | |||
| runningState | |||
| closedState | |||
| blockDbCacheSize = 256 | |||
| blockDbHandlesSize = 2000 | |||
There was a problem hiding this comment.
if handler/10 < 2000 {
blockDbHandlesSize = 1000
}else {
blockDbHandlesSize = 2000
}
Adjust this variable
fynnss
reviewed
Mar 12, 2024
fynnss
reviewed
Mar 12, 2024
jingjunLi
force-pushed
the
impl_block_store
branch
2 times, most recently
from
afc3eac to
89b7d3a
Compare
fynnss
reviewed
Mar 13, 2024
fynnss
reviewed
Mar 14, 2024
jingjunLi
force-pushed
the
impl_block_store
branch
3 times, most recently
from
3e6149f to
7e3e660
Compare
flywukong
reviewed
Mar 20, 2024
jingjunLi
force-pushed
the
impl_block_store
branch
from
8ce0dcc to
5ecce29
Compare
fynnss
reviewed
Apr 15, 2024
jingjunLi
force-pushed
the
impl_block_store
branch
2 times, most recently
from
5640825 to
288c534
Compare
* core: use finalized block as the chain freeze indicator * core/rawdb: use max(finality, head-90k) as chain freezing threshold * core/rawdb: fix tests * core/rawdb: fix lint * core/rawdb: address comments from peter * core/rawdb: fix typo
jingjunLi
force-pushed
the
impl_block_store
branch
from
288c534 to
fb15b37
Compare
zzzckck
reviewed
Apr 18, 2024
zzzckck
reviewed
Apr 18, 2024
zzzckck
reviewed
Apr 18, 2024
zzzckck
approved these changes
Apr 18, 2024
This was referenced Apr 26, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request uses the finalized block tag as the indicator for chain freezing and introduces the separation of block data into a dedicated blockdb. The principle support of PR: BEP-365: Support multi-database based on data pattern
uses the finalized block tag as the indicator for chain freezing
Originally, the latest 90K blocks will be retained in the key-value store by default, which is regarded as "non-finalized" in the context of Proof-of-work. However, in Proof-of-Stake, we do have the concrete finalized block tag from the consensus client. Therefore, it makes more sense to directly rely on the Finalized for chain freezing.
By deploying it on a full node, we can see there are only 23 blocks left in the key-value store.
introduces the separation of block data into a dedicated blockdb
recent blocks are first stored in a key-value (KV) database. Once they reach the ancient threshold, they will be moved to the ancient database, which wastes some disk bandwidth. Another thing that we should be aware of is EIP-4844. Once BSC supports EIP-4844, the block size will increase due to the inclusion of blobs. The size of the blob storage may not increase over time, but it will still demand extra disk space from the node operators.
Split database by data pattern will make disk bandwidth usage more reasonable and improve the whole performance.
Rationale
tell us why we need these changes...
Example
add an example CLI or API response...
Changes
Notable changes: