Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bonsai based reference test worldstate #5686

Merged
merged 13 commits into from
Sep 14, 2023
Merged

Bonsai based reference test worldstate #5686

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
f7d16f6
create a bonsai based reference test worldstate
garyschulte Jul 7, 2023
4b6c7a6
fix some tests and initialization
garyschulte Jul 26, 2023
4afe52f
interim commit, using Function as interface rather than PreImageStorage
garyschulte Aug 4, 2023
d9efe76
move hashing and preImage functions into a PreImageProxy class
garyschulte Aug 4, 2023
e689332
a few fixes for preImageProxy usage, account and storage streaming
garyschulte Aug 24, 2023
6659509
reference tests fix and a little cleanup
garyschulte Aug 25, 2023
ee7c22c
fix tests collisions
matkt Aug 29, 2023
30d21d2
pass limit into streamFlatAccounts, make ref test preimageproxy synch…
garyschulte Aug 29, 2023
9e6f18b
spotless vs intellij imports
garyschulte Aug 29, 2023
5c18c90
remove commented out Forest worldstate
garyschulte Aug 29, 2023
3ae8398
implement feedback from Karim, isolate BonsaiPreImageProxy, add endKe…
garyschulte Sep 1, 2023
27d1e74
filter inMemory key value storage rather than takeWhile to avoid sort
garyschulte Sep 1, 2023
a407318
rebase, javadoc and a bit of cleanup
garyschulte Sep 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix tests collisions 
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
  • Loading branch information
@matkt @garyschulte
matkt authored and garyschulte committed Sep 14, 2023
commit ee7c22ca8b6f591e378b5cec34f2289b1711490e
37 changes: 24 additions & 13 deletions ...ava/org/hyperledger/besu/ethereum/bonsai/worldview/BonsaiWorldStateUpdateAccumulator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,19 @@ public void commit() {
final BonsaiAccount updatedAccount;
final BonsaiValue<BonsaiAccount> updatedAccountValue =
accountsToUpdate.get(updatedAddress);

final Map<StorageSlotKey, BonsaiValue<UInt256>> pendingStorageUpdates =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to fullsync some nodes to be sure we are not adding a consensus issue
(goerli, sepolia, and maybe etc at the minimum)

Copy link
Contributor Author

@garyschulte garyschulte Sep 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I snap synced goerli and mainnet nodes a few days ago and it has been going well. I will start a full sync on goerli and another on sepolia and let them run over the weekend

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you try to run a mainnet snapsync with the heal flat db flag ? I see that this PR is modifying a lot the stream of the flat db datas and it will be nice if can be sure that it's ok

storageToUpdate.computeIfAbsent(
updatedAddress,
k ->
new StorageConsumingMap<>(
updatedAddress, new ConcurrentHashMap<>(), storagePreloader));

if (tracked.getStorageWasCleared()) {
storageToClear.add(updatedAddress);
pendingStorageUpdates.clear();
}

if (tracked.getWrappedAccount() == null) {
updatedAccount = new BonsaiAccount(this, tracked);
tracked.setWrappedAccount(updatedAccount);
Expand All @@ -316,6 +329,17 @@ public void commit() {
}
if (tracked.getStorageWasCleared()) {
updatedAccount.clearStorage();
wrappedWorldView()
.getAllAccountStorage(updatedAddress, updatedAccount.getStorageRoot())
.forEach(
(keyHash, entryValue) -> {
final StorageSlotKey storageSlotKey =
new StorageSlotKey(Hash.wrap(keyHash), Optional.empty());
final UInt256 value = UInt256.fromBytes(RLP.decodeOne(entryValue));
pendingStorageUpdates.put(
storageSlotKey, new BonsaiValue<>(value, null, true));
});
updatedAccount.setStorageRoot(Hash.EMPTY_TRIE_HASH);
}
tracked.getUpdatedStorage().forEach(updatedAccount::setStorageValue);
}
Expand Down Expand Up @@ -344,19 +368,6 @@ public void commit() {
return;
}

final StorageConsumingMap<StorageSlotKey, BonsaiValue<UInt256>>
pendingStorageUpdates =
storageToUpdate.computeIfAbsent(
updatedAddress,
__ ->
new StorageConsumingMap<>(
updatedAddress, new ConcurrentHashMap<>(), storagePreloader));

if (tracked.getStorageWasCleared()) {
storageToClear.add(updatedAddress);
pendingStorageUpdates.clear();
}

final TreeSet<Map.Entry<UInt256, UInt256>> entries =
new TreeSet<>(Map.Entry.comparingByKey());
entries.addAll(updatedAccount.getUpdatedStorage().entrySet());
Expand Down