Repository navigation
Optional TTL/size bound on the dedupe store + durability docs #220
Description
Activity
Corroboration from nas-observability dogfooding (jfwoods, 2026-05-11): independently flagged the same unbounded Pebble growth —
CheckAndMarkdoesSet(key, unixNano, pebble.Sync)and never deletes, so the store grows for the life of the deployment; mitigated there by wiping${NAS_DATA_ROOT}/wavehouse/pebblebetween runs. Suggested shape: a configurableWH_DEDUPE_TTLapplied as a periodic compaction/GC sweep for Pebble.(Note: an older audit also flagged the ScyllaDB/distributed dedupe backend's lack of TTL — that backend has since been removed, so embedded Pebble is the whole surface now.)
@/tmp/pr_edits/comment5826494260.md
Area: dedupe · ops — friction / docs · found via WaveHouse-Stats dogfooding
Expected: bounded dedupe memory + a clear durability contract.
Actual: an id is a duplicate forever —
CheckAndMarknever expires keys (internal/dedupe/embedded.go), so the Pebble set grows unbounded. The set is local; lose the data volume and previously-seen ids pass again. None of this is documented.Impact (Stats): reinforced keeping Pebble on a persistent volume + a
ReplacingMergeTreebackstop. Unbounded growth is accepted at our volume, but the durability/retention contract should be explicit.Scope: optional TTL or size cap, and document the durability contract (local-only, lost with the volume).
From
WAVEHOUSE-FEEDBACK.md(WaveHouse-Stats dogfooding); validated by code-read against0f8826con 2026-06-04.