Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full-Trie Mark in Parallel #1219

Merged
merged 56 commits into from
Oct 23, 2020
Merged

Full-Trie Mark in Parallel #1219

Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
1102f12
commit marks only on block boundary
RatanRSur Jun 2, 2020
fc99761
test changes
RatanRSur Jun 2, 2020
655e571
truer in order verification for case where transaction size isn't 1
RatanRSur Jun 11, 2020
8f6cfaa
flatmap
RatanRSur Jun 11, 2020
e3e749e
simplify test
RatanRSur Jun 11, 2020
1cf490a
switch back to old sweepBefore
RatanRSur Jun 11, 2020
6dae00c
remove mentions to tx
RatanRSur Jun 11, 2020
50247e0
logging and timing and exiting
RatanRSur Jun 15, 2020
6a43c1b
ignore failure and logging
RatanRSur Jun 17, 2020
3b866b2
retry on most errors, fail on a MerkleTrieException
RatanRSur Jun 17, 2020
785c287
add ops per transaction back to integration test
RatanRSur Jun 17, 2020
e24e384
spotless
RatanRSur Jun 17, 2020
29e409c
medium tx
RatanRSur Jun 17, 2020
27b5843
visit all nodes in parallel
RatanRSur Jul 14, 2020
ba521ff
moving some stuff around
RatanRSur Jul 14, 2020
a56881b
change to CompletableFuture and wait on them
RatanRSur Jul 14, 2020
c4dbca0
unlock pending marks once the marks have been added to the tx
RatanRSur Jul 14, 2020
7e5ab65
use fixed thread pool
RatanRSur Jul 14, 2020
6d41c2b
join instead of get
RatanRSur Aug 6, 2020
190afb3
make single threaded visitAll return void
RatanRSur Aug 6, 2020
9b3d8bc
report when mark work items finish
RatanRSur Aug 6, 2020
9900b92
debug -> info
RatanRSur Aug 6, 2020
c34ea60
markNodes directly to disk
RatanRSur Aug 6, 2020
6a49649
ReadWriteLock for adding nodes during full trie marking
RatanRSur Aug 7, 2020
d5cd282
make markNode and markNodes both use the same locking mechanism
RatanRSur Aug 7, 2020
cb228e0
typo
RatanRSur Aug 10, 2020
0a252f4
parallelize *storage* marking instead of whole account subtries
RatanRSur Aug 21, 2020
784aa23
mark storage tries while visiting state trie
RatanRSur Aug 31, 2020
0572790
Merge branch 'master' into pruning-parallel
RatanRSur Sep 3, 2020
224168f
caller runs if max capacity
RatanRSur Sep 8, 2020
68de7c0
bring back multiple task generators
RatanRSur Sep 17, 2020
182f6ab
Merge remote-tracking branch 'upstream/master' into pruning-parallel
RatanRSur Oct 5, 2020
26f6f8b
Merge remote-tracking branch 'upstream/master' into pruning-parallel
RatanRSur Oct 21, 2020
3b21749
rename
RatanRSur Oct 21, 2020
6dbcea7
documentation
RatanRSur Oct 21, 2020
f549a4c
static
RatanRSur Oct 21, 2020
457d469
move constants
RatanRSur Oct 21, 2020
51abfaa
undo some testing changes
RatanRSur Oct 21, 2020
e52d809
thread pool holds 16 tasks
RatanRSur Oct 21, 2020
0f8dba7
comment fixup
RatanRSur Oct 21, 2020
9aa005f
remove comment
RatanRSur Oct 21, 2020
6d22d3e
param
RatanRSur Oct 21, 2020
3b91245
log messages
RatanRSur Oct 21, 2020
08adbb3
rename lock
RatanRSur Oct 21, 2020
33d74dd
fix metrics bug
RatanRSur Oct 21, 2020
bd2b55b
rename locks
RatanRSur Oct 21, 2020
97110e9
typo
RatanRSur Oct 22, 2020
26ec138
switch to non-blocking code
RatanRSur Oct 22, 2020
5700395
comment
RatanRSur Oct 22, 2020
76367cc
Revert "switch to non-blocking code"
RatanRSur Oct 23, 2020
e1d15ad
Revert "comment"
RatanRSur Oct 23, 2020
40de52a
remove unused var
RatanRSur Oct 23, 2020
5fb2372
process root node in executor
RatanRSur Oct 23, 2020
e48fe7e
lower ops per tx
RatanRSur Oct 23, 2020
4d3bd8b
Merge remote-tracking branch 'upstream/master' into pruning-parallel
RatanRSur Oct 23, 2020
aa15d98
errata
RatanRSur Oct 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
move constants 
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
  • Loading branch information
@RatanRSur
RatanRSur committed Oct 21, 2020
commit 457d469aba4f61000b179884a551c2dce1661303
5 changes: 3 additions & 2 deletions ethereum/core/src/main/java/org/hyperledger/besu/ethereum/worldstate/MarkSweepPruner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,12 @@

public class MarkSweepPruner {

private static final int DEFAULT_OPS_PER_TRANSACTION = 50_000;
private static final Logger LOG = LogManager.getLogger();
private static final byte[] IN_USE = Bytes.of(1).toArrayUnsafe();

private static final int DEFAULT_OPS_PER_TRANSACTION = 50_000;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious about this - is it really very advantageous to bundle up changes into large transactions? Versus just picking a reasonable size and committing more frequently? I also wonder if the node were under heavy load if its possible we might hit the timeout.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable to me, I'll lower it to 10k. Let me know if you think it should go lower.

private static final int MAX_MARKING_THREAD_POOL_SIZE = 2;

private final int operationsPerTransaction;
private final WorldStateStorage worldStateStorage;
private final MutableBlockchain blockchain;
Expand All @@ -64,7 +66,6 @@ public class MarkSweepPruner {
private volatile long nodeAddedListenerId;
private final ReadWriteLock markLock = new ReentrantReadWriteLock();
private final Set<Bytes32> pendingMarks = Collections.newSetFromMap(new ConcurrentHashMap<>());
private static final int MAX_MARKING_THREAD_POOL_SIZE = 2;

public MarkSweepPruner(
final WorldStateStorage worldStateStorage,
Expand Down