Promote segmented storage #5700
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
garyschulte
force-pushed
the
feature/promote-segmented-storage
branch
6 times, most recently
from
0542b50 to
8645360
Compare
garyschulte
force-pushed
the
feature/promote-segmented-storage
branch
2 times, most recently
from
ebf9a94 to
1b65c5c
Compare
garyschulte
force-pushed
the
feature/promote-segmented-storage
branch
2 times, most recently
from
aaba707 to
8fa3b98
Compare
Signed-off-by: garyschulte <garyschulte@gmail.com>
todo: get rocks happy remove InMemoryKeyValueStorage in favor of the adapter Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
* couple segment mapping errors * wrap segmented storage transactions in a validating decorator (useless but necessary for unit test suite) * fix a few tests that were using mock segment id's Signed-off-by: garyschulte <garyschulte@gmail.com>
…ere snapshots were not cloning sub-maps Signed-off-by: garyschulte <garyschulte@gmail.com>
garyschulte
force-pushed
the
feature/promote-segmented-storage
branch
from
8fa3b98 to
b8a572f
Compare
garyschulte
commented
Jul 25, 2023
ethereum/core/src/test/java/org/hyperledger/besu/ethereum/worldstate/MarkSweepPrunerTest.java
Outdated
Show resolved
Hide resolved
Signed-off-by: garyschulte <garyschulte@gmail.com>
garyschulte
force-pushed
the
feature/promote-segmented-storage
branch
from
75b1330 to
109b0a3
Compare
daniellehrner
requested changes
Jul 26, 2023
...b/src/main/java/org/hyperledger/besu/plugin/services/storage/rocksdb/RocksDBTransaction.java
Outdated
Show resolved
Hide resolved
...eum/core/src/test/java/org/hyperledger/besu/ethereum/bonsai/BonsaiWorldStateArchiveTest.java
Outdated
Show resolved
Hide resolved
…ve test Signed-off-by: garyschulte <garyschulte@gmail.com>
daniellehrner
approved these changes
Jul 27, 2023
garyschulte
added
TeamChupa
elenduuche
pushed a commit
to elenduuche/besu
that referenced
this pull request
Aug 16, 2023
promote segmented storage to plugin-api, implement SegmentedInMemoryKeyValueStorage Signed-off-by: garyschulte <garyschulte@gmail.com>
garyschulte
added a commit
to garyschulte/besu
that referenced
this pull request
Aug 28, 2023
promote segmented storage to plugin-api, implement SegmentedInMemoryKeyValueStorage Signed-off-by: garyschulte <garyschulte@gmail.com>
garyschulte
added a commit
to garyschulte/besu
that referenced
this pull request
Aug 28, 2023
promote segmented storage to plugin-api, implement SegmentedInMemoryKeyValueStorage Signed-off-by: garyschulte <garyschulte@gmail.com>
garyschulte
added a commit
to garyschulte/besu
that referenced
this pull request
Aug 28, 2023
promote segmented storage to plugin-api, implement SegmentedInMemoryKeyValueStorage Signed-off-by: garyschulte <garyschulte@gmail.com>
eum602
pushed a commit
to lacchain/besu
that referenced
this pull request
Nov 3, 2023
promote segmented storage to plugin-api, implement SegmentedInMemoryKeyValueStorage Signed-off-by: garyschulte <garyschulte@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR description
At a high level, this PR promotes SegmentedKeyValueStorage into the plugin-api for storage.
The primary benefit of this besides addressing tech debt is that we can directly use segmented storage and atomically commit across multiple segments. This is mainly beneficial for bonsai worldstate, which comprises four separate key value store segments.
This is motivated by "unclean" worldstate storage that can result from besu app or the host machine crashing for a variety of reasons. RocksDB handles unclean shutdowns very well, but if the besu application was in the midst of a worldstate commit, each of the individual worldstate storages might be in different states of commit/rollback at the time of the crash. By having composed storage, we can rely on the guarantees of rocksdb to atomically commit or rollback changes in the event of a crash or halt, like what resulted in #5576
Fixed Issue(s)
fixes #5362 #5576