feat: filesystem-backed durable capture stores#376
Merged
Conversation
New package providing Node.js implementations of the SpoolStore and DedupeIndex interfaces from @peac/capture-core. Production-grade durable storage for the Evidence Export Path. FsSpoolStore: - Append-only JSONL with explicit commit() durability model - Hard-cap limits (maxEntries, maxFileBytes) -- SpoolFullError on breach - Single-writer lockfile guard (opt-in stale break) - Crash recovery: incomplete tail truncation, chain linkage verification - Meta file cache (spool.jsonl.meta.json) for O(1) cold start - Auto-commit timer (default 5s) for periodic fsync - Streaming reads via custom line parser (no full-file materialization) - Directory fsync on first create (best-effort) - Read-only degraded mode on corruption (export/verify still work) FsDedupeIndex: - Map-backed append-only file with last-write-wins semantics - Persist/reload across restarts, duplicate line tolerance - commit() exposed via type guard (capture-core interface unchanged) - Directory fsync on first create (best-effort) Line reader (threat model fix): - Pre-materialization maxLineBytes enforcement at Buffer level - Oversized lines never converted to JS string (prevents OOM) - Memory freed immediately when line exceeds limit - CRLF normalization (safe for cross-platform spool files) - Used by both read() and fullScan() Also: - docs/ARCHITECTURE.md version bump to 0.10.11 - scripts/check-publish-list.sh updated for 40th package - 69 tests across 5 test files (including SIGKILL durability test) - Zero external dependencies (only @peac/capture-core)
Package READMEs are consumer-facing docs where `npm install` is the standard install instruction. Add `packages/.*/README\.md` to the NPM_ALLOW pattern so guard.sh does not flag them.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Remove unused imports (FsDedupeIndex, FsSpoolStore, SpoolFullError) and unused variables (giantSize, store) flagged by code quality bot.
jithinraj
changed the title
7 tasks
jithinraj
added a commit
that referenced
this pull request
Feb 15, 2026
The CI publish-manifest closure check failed because @peac/capture-node was not in the manifest or directory mappings. This package was added in PR #376 but the publish-related scripts were not updated. - publish-manifest.json: add capture-node (20 -> 21 packages) - check-publish-closure.ts: add capture-node nested mapping - pack-smoke.mjs: add capture-node PKG_DIRS mapping
jithinraj
added a commit
that referenced
this pull request
Feb 15, 2026
* feat(adapter-openclaw): plugin activation bridge and keygen utility Add activate() as the single canonical entry point that wires all adapter components from config. Calls existing factory functions (resolveSigner, createPluginInstance, createFileReceiptWriter) with FsSpoolStore and FsDedupeIndex from @peac/capture-node -- no logic duplication. Add generateSigningKey() for Ed25519 keypair generation via @peac/crypto, with keygenCli() for CLI usage and peac-keygen bin entry. Changes: - activate.ts: high-level entry wiring stores, signer, writer, instance, tools - keygen.ts: Ed25519 key generation, JWK file output with 0o600 permissions - keygen-cli.ts: CLI entry point for peac-keygen binary - index.ts: export activate, keygen types and functions - package.json: add @peac/capture-node dep, bin entry, bump @types/node to ^22 - tsup.config.ts: add keygen-cli entry point - openclaw.plugin.json: bump version 0.10.6 -> 0.10.12 - check-publish-list.sh: add adapter-openclaw to tested, capture-core to tracked, fix hardcoded count 40 -> 42 Tests: 22 new tests (12 activate + 10 keygen), 161 total adapter tests passing. * fix: add @peac/capture-node to publish manifest and directory mappings The CI publish-manifest closure check failed because @peac/capture-node was not in the manifest or directory mappings. This package was added in PR #376 but the publish-related scripts were not updated. - publish-manifest.json: add capture-node (20 -> 21 packages) - check-publish-closure.ts: add capture-node nested mapping - pack-smoke.mjs: add capture-node PKG_DIRS mapping * fix: address PR review findings - createStatusTool: accept getter function for live stats instead of stale snapshot. Backwards-compatible: still accepts static PluginStats. - activate.ts: pass stats getter to createStatusTool - activate.ts: add explicit type annotation on onWarning callback - tsconfig.core.json: add packages/capture/node/src to includes so CI typecheck resolves @peac/capture-node imports * fix: add @peac/capture-node path mapping to tsconfig.base.json typecheck:core failed because the paths block was missing capture-node, so TS could not resolve the bare specifier imported in adapter-openclaw/src/activate.ts. * chore: align plugin manifest version + add duplicate JSON key guard - openclaw.plugin.json version 0.10.12 -> 0.10.11 to match package.json - Add scripts/check-json-dupes.mjs: detects duplicate keys in JSON files (JSON parsers silently accept duplicates with last-write-wins) - Wire check-json-dupes into guard.sh alongside existing bidi scanner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New package
@peac/capture-nodeproviding Node.js implementations of theSpoolStoreandDedupeIndexinterfaces from@peac/capture-core. Production-grade durable storage for the Evidence Export Path.commit()durability, hard-cap limits, single-writer lockfile, crash recovery (tail truncation + chain verification), meta file cache for O(1) cold start, auto-commit timer, streaming reads, directory fsync on first create, read-only degraded mode on corruptioncommit()via type guard (capture-core interface unchanged), directory fsync on first createmaxLineBytesat Buffer level before string materializationThreat model fix
maxLineBytesis enforced pre-materialization in bothread()andfullScan(). The custom streaming line parser (line-reader.ts) operates on rawBufferchunks -- oversized lines are never converted to JS strings, preventing OOM from a single large line in a spool file. Memory is freed immediately when a line exceeds the limit, andaccumulatedBytesis capped to prevent counter overflow on pathological files.This replaces
readline(which materializes full strings before any size check) andfs.readFile()(which loads the entire file).Corruption boundaries
onWarningfiredprev_entry_digestchain failedmaxLineBytes)When corrupt, the spool enters read-only mode: new captures blocked, export/verify/query still work.
Durability contract
append()writes to OS page cache (no fsync) -- fast, not crash-safe alonecommit()calls fsync -- the explicit durability pointcommit()periodically when dirtyDependency audit triage
6 vulnerabilities found in monorepo (2 high, 3 moderate, 1 low). None affect
@peac/capture-node(zero external deps). Findings are insurfaces/*,examples/*, and one low-severity transitive in@peac/middleware-express. Will be addressed via dependency bumps.pages deploysurfaces/workers/cloudflare(dev-only)surfaces/nextjs/middleware(dev/example)surfaces/nextjs/middlewareexamples/+@peac/middleware-expressTest plan
pnpm lint,pnpm typecheck:core,pnpm format:check-- all cleanscripts/guard.sh,scripts/check-planning-leak.sh,scripts/check-publish-list.sh-- all passFiles
New package:
packages/capture/node/src/fs-spool-store.tssrc/fs-dedupe-index.tssrc/line-reader.tssrc/lockfile.tssrc/errors.tssrc/index.tstests/fs-spool-store.test.tstests/fs-dedupe-index.test.tstests/lockfile.test.tstests/integration.test.tstests/line-reader.test.tstests/fixtures/spool-crash-writer.tsREADME.mdModified
docs/ARCHITECTURE.mdpnpm-lock.yamlscripts/guard.shnpm installin package READMEs (consumer-facing docs)scripts/check-publish-list.sh