Switch RocksDB block cache from LRU to HyperClockCache#4473
Open
AhmedSoliman wants to merge 10 commits intomainfrom
Open
Switch RocksDB block cache from LRU to HyperClockCache#4473AhmedSoliman wants to merge 10 commits intomainfrom
AhmedSoliman wants to merge 10 commits intomainfrom
Conversation
This was referenced Mar 9, 2026
Extract two generic, reusable utilities into restate-futures-util: **monotonic_token**: A lightweight mechanism for a producer to signal completion of a prefix of sequentially issued work items. Provides Token<T>, TokenOwner<T>, Tokens<T>, and TokenListener<T> types with a phantom type parameter to prevent mixing tokens from different domains. Uses atomics (Relaxed/Release/Acquire) for lock-free operation — no RwLock or watch overhead. **waiter_queue**: A priority-drainable queue (WaiterQueue<K, V>) designed for the common case where entries arrive in key-order. Uses an adaptive strategy: push_back for in-order inserts (O(1)), binary-search insert for out-of-order (rare). Drain is always a simple front-pop. Includes a Criterion benchmark comparing four strategies (naive, compact, adaptive, sorted-insert). Both modules include comprehensive documentation and tests. Neither references any specific use-case — they are general-purpose building blocks.
…closed This makes turning off loglet workers cleaner (next PR).
- Priority-queue based writer allowing seal messages to jump the queue - Deduplication of seal messages and store messages - Improved metrics for the write path (counting bytes, stores, and store status) - Loglet workers will shutdown when quiescent and release resources - Writer task limits the batch based on the memtable size as a reasonable guidance and removing the need for `write-batch-commit-count` config. - Removed the returned WriteBatch in the error case since write errors are terminal. This reduces the size of the returned Result.
Replace raw usize/NonZeroUsize types with type-safe NonZeroByteCount for all RocksDB memory budget configurations across the codebase. Key changes: - CommonOptions: make rocksdb_total_memory_size private behind a getter that enforces a 256 MiB minimum; rename rocksdb_actual_total_memtables_size to rocksdb_total_memtables_size with a 32 MiB floor; remove the 5% safety margin (rocksdb_safe_total_memtables_size); clamp memtables ratio to [0.1, 1.0] instead of [0.0, 1.0] - LogServerOptions: remove data_service_memory_limit config (memory pool capacity is now derived from rocksdb_data_memtables_budget); fix metadata memtables budget to a constant 8 MiB instead of a ratio; enforce 40 MiB (32 MiB data + 8 MiB metadata) minimum for log-server memory budget - MetadataServerOptions/StorageOptions: change rocksdb_memory_budget return types from usize to NonZeroByteCount with per-component minimums - ByteCount: add arithmetic ops (Add, Mul, saturating_add/mul), Default, and TryFrom<u64> for NonZeroByteCount - Remove unnecessary runtime assertions that were checking for non-zero on already non-zero types
Move db-level properties (is-write-stopped, background-errors, num-running-compactions, actual-delayed-write-rate) from the per-CF set to the per-DB set since they are database-wide. Also fix the unit of actual-delayed-write-rate to Bytes and add blob-db metrics (live-blob-file-size, live-blob-file-garbage-size) and obsolete-sst-files-size for log-server observability.
…ion budgets Remove the shared rocksdb_max_background_jobs config (which gave every database CPU_COUNT background jobs) and replace it with role-aware per-database budgets for flushes and compactions. Flushes (latency-critical) are split equally across databases while compactions (throughput-heavy) are weighted ~65% toward the partition-store. Metadata-server and local-loglet get a fixed budget of 1+1. Also adds worker.snapshots.export-concurrency-limit (default 4) to replace the snapshot export concurrency that was previously derived from max_background_jobs.
Add two new options to RocksDbOptions: - rocksdb-disable-wal-compression: disables Zstd WAL compression (default: false) - rocksdb-disable-l0-l1-compression: disables Zstd L0/L1 SST compression (default: false) Both options cascade from common rocksdb config and default to compression enabled, preserving existing behavior. A new build_compression_per_level() helper in the rocksdb crate constructs per-level compression arrays.
HyperClockCache (HCC) is the recommended default for RocksDB, offering better scalability under concurrent access compared to LRU cache.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HyperClockCache (HCC) is the recommended default for RocksDB, offering
better scalability under concurrent access compared to LRU cache.
Stack created with Sapling. Best reviewed with ReviewStack.