Repository navigation
Expose RocksDB WriteBufferManager with cost-to-cache support - #584
Merged
Merged
Conversation
Adds a process-wide WriteBufferManager singleton in DBSettings, lazy- created on first DB open and shared across every database in the process. Memtable memory is bounded by a single configurable limit rather than each DB managing its own budget. New `RocksDatabase.config()` options: - `writeBufferManagerSize` — bytes of memtable memory the WBM caps across all DBs (0 disables); runtime-mutable via SetBufferSize - `writeBufferManagerCostToCache` — when true, memtable charges show up in `block-cache-usage` as pinned entries, giving a single observability metric for read cache + memtable footprint - `writeBufferManagerAllowStall` — when true, writes stall once the cap is reached instead of letting memtables grow past it Implementation notes: - DBSettings::getInstance() now uses C++11 magic statics for thread- safe lazy initialization (previously raced on concurrent first-call from libuv worker threads). - Configuration fields are `std::atomic` so JS-thread Config() writes are safe against worker-thread reads in getWriteBufferManager(). - WBM creation is mutex-protected; double-checked locking on the atomic size guards against teardown between load and lock. - `costToCache` and `allowStall` are fixed at first WBM creation (already-open DBs hold shared_ptr refs; the WBM can't be swapped). Config() writes after creation are ignored with an stderr warning. - Setting size=0 at runtime is a "no new attachments" signal, not a teardown — RocksDB asserts SetBufferSize(>0). Verified by 11 tests covering: negative-size rejection, default-off path, costToCache wiring, multi-DB sharing, runtime resize, active memtable reclamation after flush, block-cache-usage charge visibility, the actual structural cap (writeBufferManagerSize, not blockCacheSize), and maintain-window cycling. Co-Authored-By: Claude Sonnet 4.7 <noreply@anthropic.com>
Contributor
📊 Benchmark Resultsget-sync.bench.tsgetSync() > random keys - small key size (100 records)
getSync() > sequential keys - small key size (100 records)
ranges.bench.tsgetRange() > small range (100 records, 50 range)
realistic-load.bench.tsRealistic write load with workers > write variable records with transaction log
transaction-log.bench.tsTransaction log > read 100 iterators while write log with 100 byte records
Transaction log > read one entry from random position from log with 1000 100 byte records
worker-put-sync.bench.tsputSync() > random keys - small key size (100 records, 10 workers)
worker-transaction-log.bench.tsTransaction log with workers > write log with 100 byte records
Results from commit 3a404d1 |
…stdio Don't write warnings to stderr
cb1kenobi
approved these changes
May 26, 2026
| instance = std::unique_ptr<DBSettings>(new DBSettings()); | ||
| } | ||
| return *instance; | ||
| static DBSettings instance; |
|
|
||
| // costToCache can't be retro-applied; warn if the user attempts it. | ||
| if (wbmAlreadyCreated && costToCacheProvided && newCostToCache != oldCostToCache) { | ||
| std::fprintf(stderr, |
Member
There was a problem hiding this comment.
A library should not write to stdio. Instead, we can throw and the user can retry the config() call. In order to throw, we need to reorder this function as to do this check before mutating any state. I've got a PR that targets the branch to show what I'm talking about: #585.
This was referenced May 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a process-wide
WriteBufferManagersingleton inDBSettings, lazy-created on first DB open and shared across every database in the process. Memtable memory is bounded by a single configurable limit rather than each DB managing its own budget. WithcostToCache: true, memtable charges show up as pinned entries in the block cache, giving a single observability metric for read cache + memtable footprint.Why
In Harper Fabric's shared-host deployment, each container opens many RocksDB databases (one per Harper user database/system DB). Memtable memory accumulates per-DB and per-column-family — for a customer like Canopy with 14 tables, the maintain window alone (
max_write_buffer_size_to_maintain× CFs) can account for ~1.8 GB of resident anonymous memory. Today there's no process-level cap. WriteBufferManager fixes that.What this changes
New
RocksDatabase.config()options:writeBufferManagerSize00disables. Runtime-mutable viaSetBufferSize.writeBufferManagerCostToCachefalsewriteBufferManagerAllowStallfalseSetAllowStall.Implementation notes
DBSettings::getInstance()now uses C++11 magic statics — the previousunique_ptrlazy-init raced on concurrent first-call from libuv worker threads. Independent of WBM, this also fixes a pre-existing risk.std::atomicso JS-threadConfig()writes are safe against worker-thread reads ingetWriteBufferManager().costToCacheis fixed at first creation (already-open DBs holdshared_ptrrefs; the cache reservation manager is configured at construction). Subsequent attempts to change it emit a stderr warning.allowStallIS updateable on a live WBM viaSetAllowStall—Config()propagates runtime changes.writeBufferManagerSize = 0at runtime means "no new attachments," not teardown. RocksDB assertsSetBufferSize(new_size > 0).What's verified by tests
The 11 tests cover the structural claims and document the limitations:
RangeErrorSetBufferSizedoesn't disrupt writescur-size-active-mem-tabledrops by 10xcostToCache, memtable charges show up inblock-cache-usage(≥1 MB delta for 2 MB writes)writeBufferManagerSize, NOTblockCacheSizeHonest caveats (covered in test comments)
Cross-model review findings addressed
Gemini — flagged the singleton race in
getInstance()(fixed: magic statics) and data races on the WBM config fields (fixed:std::atomic+ mutex). Suggested per-DB opt-out which is left for a follow-up.Codex — caught that
allowStallchanges after WBM creation were silently dropped (fixed: now propagates viaSetAllowStall).Related
Complements #575 (per-CF write buffer options) — that PR exposes the per-CF knobs, this one adds the process-wide cap that makes them safe to tune up.
Generated by Claude. — Claude Sonnet 4.7