research(nightly): structural-time keyframe retention for agent memory (ADR-346)

[ruvnet]

Summary

Nightly research run (2026-09-06). Connects two previously-unconnected crates — ruvector-agent-memory (agent memory compaction) and emergent-time (calculus of emergent/internal time) — to fix a real blind spot in CoherencePolicy's recency signal: it scores recency from a per-event logical-clock tick, which is functionally identical to emergent-time's own WallClock null hypothesis, and therefore cannot distinguish a burst of redundant churn (retries, duplicate re-observations) from a burst of genuinely new information.

Hypothesis: on a 360-memory synthetic corpus (40 "regime-shift" topics, each followed by 8 near-duplicate "churn" memories), replacing tick-based recency with emergent-time's Structural Proper Time clock should preserve regime-shift memories through aggressive compaction better than tick recency, without hurting steady-state recall or costing an unreasonable amount of compute.

What was found, honestly, including a negative result:

• Candidate B1 (turn cumulative structural time into a [0,1] rank score, mirroring how CoherencePolicy already ranks by tick count) does not work: measured +0.0pp over baseline. Cumulative structural time is monotone non-decreasing in insertion order — exactly like the tick count it replaces — so ranking by it barely changes the top-K set. This is retained in the tree as tested, documented reference code, not deleted.
• Candidate B2 (use emergent_time::structural_clock::keyframes — the primitive that crate already ships for budget-constrained trajectory sampling, applied here to retention instead of compression) works: +25.0pp regime-shift-memory survival over the tick-recency baseline, +22.5pp over a fair, dependency-free cheap competitor (DedupGatedRecency, included specifically to rule out a strawman win), -0.5pp Recall@10 (well within the 3pp tolerance), and 2.5-3.3x compaction overhead (not the 1,800x+ seen in the prior nightly's mincut-gated approach for a related problem).

Acceptance: ACCEPT (all pre-declared mandatory gates pass for candidate B2; candidate B1's failure is reported, not hidden).

Architecture

New module ruvector-agent-memory::structural_recency, feature-gated behind structural-time (optional path dependency on emergent-time), off by default. No change to CoherencePolicy's default behavior or any existing public API.

flowchart LR
    Insertion["40 regime-shifts x (1 signal + 8 churn)"] --> Traj["StateSnapshot trajectory\n(embedding Δv, coherence ΔC)"]
    Traj --> Clock["emergent_time::StructuralProperTime"]
    Clock --> Score["rank score -> StructuralTimeRecency (B1)"]
    Clock --> KF["keyframes() budget sample -> StructuralKeyframeRetention (B2)"]
    Score -.->|"+0.0pp vs baseline"| Neg["negative result, retained"]
    KF -->|"+25pp vs baseline, +22.5pp vs fair baseline"| Pos["promoted, feature-gated"]

Loading

Files changed

• crates/ruvector-agent-memory/src/structural_recency.rs — DedupGatedRecency (fair baseline, always compiled), StructuralTimeRecency (B1, negative result), StructuralKeyframeRetention (B2, promoted), unit tests including a deterministic exact-duplicate-churn corpus that isolates the tie-break mechanics.
• crates/ruvector-agent-memory/examples/structural_time_recency_bench.rs — the two-experiment benchmark with pre-declared acceptance gates.
• crates/ruvector-agent-memory/{Cargo.toml,src/lib.rs} — structural-time feature wiring.
• docs/adr/ADR-346-structural-time-keyframe-agent-memory-retention.md
• docs/research/nightly/2026-09-06-structural-time-agent-memory/{README.md,gist.md} — full methodology, raw evidence, ecosystem-fit analysis (RVF/RVM/ruFlo/MCP/WASM), practical/long-horizon applications, falsification criteria.
• docs/adr/INDEX.md — regenerated via node scripts/adr-index.mjs.

Benchmark command & real results

cargo run --release -p ruvector-agent-memory --example structural_time_recency_bench --features structural-time

Experiment 1 (recency-only ablation, 360 → 40 entries):

Policy	Survival rate
CoherenceWeighted (baseline)	10.0%
DedupGatedRecency (fair baseline)	12.5%
StructuralTimeRecency (B1)	10.0%
StructuralKeyframeRetention (B2)	35.0%

Experiment 2 (production weights, 360 → 180, Recall@10):

Policy	Recall@10
CoherenceWeighted (baseline)	100.0%
DedupGatedRecency	100.0%
StructuralKeyframeRetention (B2)	99.5%

Determinism verified across 3 independent cargo run process invocations (bit-identical survival/recall/PASS-FAIL). Full raw output in the nightly README.

Darwin result

Not run this nightly — this is a two-candidate hand-designed comparison (rank-score vs. keyframe-sampling integration strategies), not a bounded evolutionary search over a parameter space. Both candidates and the fair baseline are retained in the lineage (B1 as a documented negative result, per the process's evidence-retention rule).

Flywheel result

Recorded as this PR + ADR-346 + the nightly research doc: hypothesis, sources (the two crates' existing code and their own test/benchmark cultures), selected architecture (keyframe sampling), rejected alternative (score-based ranking, with the structural reason it fails), benchmark config/output, acceptance result, and the reusable lesson (monotone clocks make bad rank scores, good samplers) for future nightlies.

Security review

No new attack surface: pure in-memory scoring over caller-provided vectors, no I/O, no new serialization, no interaction with the ledger/witness/proof-gate machinery. cargo clippy -p ruvector-agent-memory --lib --features structural-time -- -D warnings is clean.

Main limitations

• Synthetic stress-test corpus only; no real-agent-trace validation yet (listed as an explicit open question in the ADR).
• Only 2 of StructuralMetric's 5 channels are exercised (embedding, coherence); entropy/graph/prediction-error are honestly left at zero rather than fabricated.
• A degenerate exact-duplicate-trailing-tail corner case in the budget-trimming logic is documented and unit-tested but not fully resolved.
• Not benchmarked head-to-head against the prior nightly's mincut-gated-forgetting approach (ADR-345) — related but distinct failure modes, compared only qualitatively on cost.

Production recommendation

Opt-in via the structural-time Cargo feature for agent-memory deployments with known bursty/redundant activity patterns (tool-call retries, repeated confirmations). Not recommended for promotion to CoherencePolicy's default recency term without real-trace validation.

Test plan

• cargo build -p ruvector-agent-memory (with and without structural-time)
• cargo test -p ruvector-agent-memory --features structural-time --lib — 34/34 passing
• cargo clippy -p ruvector-agent-memory --lib --features structural-time -- -D warnings — clean
• cargo fmt -p ruvector-agent-memory -- --check — clean
• cargo run --release -p ruvector-agent-memory (existing bench) — no regression, still PASSED
• cargo run --release -p ruvector-agent-memory --example structural_time_recency_bench --features structural-time — ACCEPT, reproducible across 3 independent process runs

---

🤖 Generated with claude-flow

https://claude.ai/code/session_01EgEEWJmo3ecuZobd8xr72B

---

Generated by Claude Code

[ruvnet]

CI triage: all checks are green except Tests (core-and-rest), which shows cancelled (not failed) after a ~4h run.

This is not this PR's failure: the same "Workspace CI" workflow shows an overall cancelled conclusion on the last 5 consecutive pushes to main itself (runs #1393, #1391, #1389, #1387, #1385 — all cancelled), unrelated to this PR's diff. This matches this repository's existing CI concurrency/cost-optimization behavior (a CI/CD Cost Optimization workflow is also configured), not a regression introduced here. This PR's change is a single, isolated, feature-gated addition to ruvector-agent-memory (off by default) with its own fast, deterministic local test run (34/34 passing in ~0.1s) and a clean cargo clippy -D warnings — no code path touched by this diff plausibly explains a 4-hour hang in an unrelated workspace-wide test shard.

No action taken; will re-check if a future CI run on this branch shows a genuine (non-cancelled) failure in code this PR touches.

---

Generated by Claude Code