Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize RocksDB WAL file #6328

Merged
merged 4 commits into from
Jan 4, 2024
Merged

Optimize RocksDB WAL file #6328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0d537bb
Optimize RocksDB WAL file
fab-10 Dec 18, 2023
90af6b0
Update CHANGELOG
fab-10 Dec 22, 2023
7a3dd78
Merge branch 'main' into rocksdb-wal-opt
fab-10 Jan 4, 2024
97c913f
Merge branch 'main' into rocksdb-wal-opt
fab-10 Jan 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
- Set Ethereum Classic mainnet activation block for Spiral network upgrade [#6267](https://github.com/hyperledger/besu/pull/6267)
- Add custom genesis file name to config overview if specified [#6297](https://github.com/hyperledger/besu/pull/6297)
- Update Gradle plugins and replace unmaintained License Gradle Plugin with the actively maintained Gradle License Report [#6275](https://github.com/hyperledger/besu/pull/6275)
- Optimize RocksDB WAL files, allows for faster restart and a more linear disk space utilization [#6328](https://github.com/hyperledger/besu/pull/6328)

### Bug fixes

Expand Down
7 changes: 7 additions & 0 deletions ...ledger/besu/plugin/services/storage/rocksdb/segmented/RocksDBColumnarKeyValueStorage.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,11 @@ public abstract class RocksDBColumnarKeyValueStorage implements SegmentedKeyValu
protected static final long ROCKSDB_BLOCKCACHE_SIZE_HIGH_SPEC = 1_073_741_824L;
/** RocksDb memtable size when using the high spec option */
protected static final long ROCKSDB_MEMTABLE_SIZE_HIGH_SPEC = 1_073_741_824L;
/** Max total size of all WAL file, after which a flush is triggered */
protected static final long WAL_MAX_TOTAL_SIZE = 1_073_741_824L;
/** Expected size of a single WAL file, to determine how many WAL files to keep around */
protected static final long EXPECTED_WAL_FILE_SIZE = 67_108_864L;

/** RocksDb number of log files to keep on disk */
private static final long NUMBER_OF_LOG_FILES_TO_KEEP = 7;
/** RocksDb Time to roll a log file (1 day = 3600 * 24 seconds) */
Expand Down Expand Up @@ -237,6 +242,8 @@ private void setGlobalOptions(final RocksDBConfiguration configuration, final St
options
.setCreateIfMissing(true)
.setMaxOpenFiles(configuration.getMaxOpenFiles())
.setMaxTotalWalSize(WAL_MAX_TOTAL_SIZE)
.setRecycleLogFileNum(WAL_MAX_TOTAL_SIZE / EXPECTED_WAL_FILE_SIZE)
.setStatistics(stats)
.setCreateMissingColumnFamilies(true)
.setLogFileTimeToRoll(TIME_TO_ROLL_LOG_FILE)
Expand Down
Loading