-
Notifications
You must be signed in to change notification settings - Fork 863
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add --X-trie-log subcommand #6303
Changes from 1 commit
16c0a49
7dd4928
bf2b098
e67ae51
9b4e0c9
0b9fe83
426848e
7401b59
1b7fb72
11e6b05
f2d01e2
04f1aaa
2f01c5a
56e4c8e
78561b0
42c72cf
9961fc2
9389540
e3d4fbc
c7144fe
20b0ba5
e7d175c
b214bf2
aa75348
2bc0732
deec021
5d7b68e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
…emory Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -77,21 +77,21 @@ static void prune( | |
} | ||
|
||
// retrieve the layersToRetains hashes from blockchain | ||
final List<Hash> hashesToRetain = new ArrayList<>(); | ||
final List<Hash> trieLogKeys = new ArrayList<>(); | ||
|
||
for (long i = chainHeight; i > lastBlockToRetainTrieLogsFor; i--) { | ||
final Optional<BlockHeader> header = blockchain.getBlockHeader(i); | ||
header.ifPresent(blockHeader -> hashesToRetain.add(blockHeader.getHash())); | ||
header.ifPresent(blockHeader -> trieLogKeys.add(blockHeader.getHash())); | ||
} | ||
|
||
IdentityHashMap<byte[], byte[]> trieLogsToRetain; | ||
|
||
// TODO: maybe stop the method here if we don't find enough hashes to retain | ||
if ((long) hashesToRetain.size() == layersToRetain) { | ||
if ((long) trieLogKeys.size() == layersToRetain) { | ||
trieLogsToRetain = new IdentityHashMap<>(); | ||
// save trielogs in a flatfile as a fail-safe | ||
// save trielogs in a flatfile in case something goes wrong | ||
out.println("Obtaining trielogs to retain..."); | ||
hashesToRetain.forEach( | ||
trieLogKeys.forEach( | ||
hash -> { | ||
rootWorldStateStorage | ||
.getTrieLog(hash) | ||
|
@@ -100,30 +100,36 @@ static void prune( | |
out.println("Saving trielogs to retain in file..."); | ||
saveTrieLogsInFile(trieLogsToRetain); | ||
} else { | ||
// try to read the triLogs from the flatfile | ||
// in case something went wrong and we already pruned trielogs | ||
// users can re-un the subcommand and we will read trielogs from file | ||
trieLogsToRetain = readTrieLogsFromFile(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't understand how reading from the file helps in this case. We didn't find all the hashes but also haven't created the file at this point either unless we are relying running the command again to read the previous state There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is a failsafe in for a subsequent execution where something went terribly wrong during the truncation and restore-from-file process. what would be nice, but not idiomatic, is if we had a few "are you sure?" or "do you want to?" prompts that could be bypassed by |
||
} | ||
out.println("Clear trielogs..."); | ||
// clear trielogs storage | ||
// TODO: Add a check to ensure we have trieLogsToRetain.size() == layersToRetain | ||
rootWorldStateStorage.clearTrieLog(); | ||
|
||
// get an update and insert the trielogs we retained | ||
if (trieLogsToRetain.size() == layersToRetain) { | ||
out.println("Clear trielogs..."); | ||
rootWorldStateStorage.clearTrieLog(); | ||
out.println("Restoring trielogs retained into db..."); | ||
recreateTrieLogs(rootWorldStateStorage, trieLogsToRetain); | ||
} | ||
if (rootWorldStateStorage.streamTrieLogKeys(layersToRetain).count() == layersToRetain) { | ||
out.println("Prune ran successfully. Deleting file..."); | ||
deleteTrieLogFile(); | ||
out.println("Enjoy some GBs of storage back!..."); | ||
gfukushima marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} else { | ||
out.println("Prune failed. Re-run the subcommand to load the trielogs from file."); | ||
} | ||
} | ||
|
||
private static void recreateTrieLogs( | ||
final BonsaiWorldStateKeyValueStorage rootWorldStateStorage, | ||
final IdentityHashMap<byte[], byte[]> trieLogsToRetain) { | ||
var updater = rootWorldStateStorage.updater(); | ||
out.println("restore trielogs retained into db..."); | ||
|
||
trieLogsToRetain.forEach( | ||
(key, value) -> { | ||
updater.getTrieLogStorageTransaction().put(key, value); | ||
}); | ||
updater.getTrieLogStorageTransaction().commit(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If this fails then do have a way of restoring the trielog from the backup? |
||
|
||
if (rootWorldStateStorage.streamTrieLogKeys(layersToRetain).count() == layersToRetain) { | ||
out.println("Prune ran successfully. Deleting file..."); | ||
deleteTrieLogFile(); | ||
} else { | ||
out.println("Prune failed. Please check the logs for more details."); | ||
} | ||
out.println("Enjoy some GBs of storage back!..."); | ||
} | ||
|
||
private static void validatePruneConfiguration(final DataStorageConfiguration config) { | ||
|
@@ -220,7 +226,9 @@ private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOE | |
|
||
private static void deleteTrieLogFile() { | ||
File file = new File(trieLogFile); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When we create the file we could use the File.deleteOnExit so we don't need to worry about manually deleting this file |
||
file.delete(); | ||
if (file.exists()) { | ||
file.delete(); | ||
} | ||
} | ||
|
||
static void printCount(final PrintWriter out, final TrieLogCount count) { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure how the fail-safe is being used. Expected to see that if we got an error clearing and inserting entries we ready from the file or allow specifying an existing backup file to restore from.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a comment and instructions that will be logged for the user. Basically the idea here is, we retain the logs in a file in case something goes wrong.
In that case the user can re-run the subcommand and we will read the trielogs from the file since the column family might have been dropped already.