fix: stabilize playbook aggregation clustering

[yyiilluu]

Summary

• normalize low-signal wording from trigger text whenever a new embedding is encoded, without backfilling existing vectors
• use embedding-model-aware clustering defaults and cap HDBSCAN neighbor reach to reduce noisy clusters
• paginate aggregation candidates with a stable descending-ID keyset and frozen high-watermark, while retaining legacy offset behavior for external callers
• filter aggregation candidates consistently, and create a default aggregation config when none is stored
• add regression coverage for normalization, clustering, storage, mutation-safe pagination, and aggregation bookmark behavior

Test plan

• uv run ruff check open_source/reflexio
• uv run pyright
• OSS unit/integration suite: 5,145 passed, 13 skipped, 154 deselected, 6 subtests passed
• focused post-rebase pagination and storage suite: 138 passed
• targeted aggregation and embedding tests: 457 passed
• manual SQLite embedding proof confirmed a new trigger is encoded from normalized trigger text

Notes

Trigger normalization is intentionally prospective. Existing stored playbook embeddings are not rewritten or backfilled by this change.

Summary by CodeRabbit

• New Features

  • Playbook aggregation is now enabled by default, with safe backward-compatible defaults.
  • Clustering similarity defaults to the selected embedding model when omitted.
  • Playbook embedding generation now uses a trigger-aware embedding text helper.
  • Aggregation pagination now exhaustively reads large playbook sets without truncation.

• Bug Fixes

  • Prevented spurious clusters from playbooks missing triggers or embeddings.
  • Legacy configs using explicit null aggregation settings now migrate to defaults.
  • Stale embedding and expanded-term fields are cleared when rebuilt without source text.

[coderabbitai]

📝 Walkthrough

Walkthrough

Playbook aggregation now defaults its configuration, resolves clustering thresholds per embedding model, sanitizes trigger text for embeddings, reads storage pages exhaustively, and excludes unclusterable playbooks. Tests cover configuration migration, embedding behavior, pagination, clustering, and stale-field clearing.

Changes

Playbook aggregation pipeline

Layer / File(s)	Summary
Aggregation configuration defaults docs/lib/config-schema.ts, reflexio/models/config_schema.py, tests/models/*, reflexio/integrations/openclaw/...	Aggregation configuration is now defaulted and non-null, legacy null values migrate to defaults, and omitted clustering similarity resolves from the embedding model.
Trigger embedding and model thresholds reflexio/server/services/embedding_text.py, reflexio/server/services/storage/sqlite_storage/..., tests/server/services/test_embedding_text.py, tests/server/services/storage/*, tests/e2e_tests/conftest.py	Playbook embeddings use sanitized triggers without content fallback; storage paths clear missing-text fields and use the shared embedding helper.
Paginated aggregation snapshot reflexio/server/services/playbook/components/aggregator.py, reflexio/server/services/playbook/components/aggregator_clustering.py, reflexio/server/services/storage/storage_base/playbook/*, reflexio/server/services/storage/sqlite_storage/playbook/*	Aggregation reads all playbook pages, preserves a high-watermark, resolves model-specific clustering thresholds, skips missing embeddings and triggers, and updates HDBSCAN threshold handling.
Aggregation and storage validation tests/server/services/playbook/*, tests/server/services/storage/test_storage_contract_playbook.py	Tests validate pagination, high-watermark state, model-aware clustering, HDBSCAN parameters, outlier handling, and missing embedding or trigger behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PlaybookAggregator
  participant SQLiteStorage
  participant EmbeddingPolicy
  participant HDBSCAN
  PlaybookAggregator->>SQLiteStorage: read playbooks across bounded pages
  SQLiteStorage-->>PlaybookAggregator: return complete snapshot
  PlaybookAggregator->>EmbeddingPolicy: resolve clustering similarity
  EmbeddingPolicy-->>PlaybookAggregator: return model threshold
  PlaybookAggregator->>HDBSCAN: cluster embedded playbooks
  HDBSCAN-->>PlaybookAggregator: return clusters

Loading

Possibly related PRs

• ReflexioAI/reflexio#249: Both changes modify PlaybookAggregator orchestration and initialization.
• ReflexioAI/reflexio#262: Both changes affect playbook storage mixins and pagination interfaces.
• ReflexioAI/reflexio#270: Both changes modify the playbook clustering stack.

Suggested reviewers: guangyu-reflexio

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 35.85% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the PR’s main focus on stabilizing playbook aggregation clustering.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-org48-playbook-aggregation

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@reflexio/server/services/playbook/components/aggregator.py`:
- Around line 65-75: Update _read_all_pages and its get_*_playbooks callers to
replace mutable OFFSET pagination with a consistent keyset cursor plus
high-watermark, or reuse a backend-consistent read snapshot. Ensure each page
request preserves the original ordered result set across inserts, archives, and
deletions while retaining exhaustive aggregation and bookmark behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 08d747aa-98bf-419a-bb47-bd4482dbab31

📥 Commits

Reviewing files that changed from the base of the PR and between a8e2c14 and ae655ff.

⛔ Files ignored due to path filters (1)

• tests/test_data/mock_playbooks.csv is excluded by !**/*.csv

📒 Files selected for processing (18)

• docs/lib/config-schema.ts
• reflexio/integrations/openclaw/plugin/src/openclaw_smart/publish.py
• reflexio/models/config_schema.py
• reflexio/server/services/embedding_text.py
• reflexio/server/services/playbook/components/aggregator.py
• reflexio/server/services/playbook/components/aggregator_clustering.py
• reflexio/server/services/storage/sqlite_storage/governance/_rebuild_hide.py
• reflexio/server/services/storage/sqlite_storage/playbook/_agent.py
• reflexio/server/services/storage/sqlite_storage/playbook/_user.py
• tests/e2e_tests/conftest.py
• tests/models/test_playbook_aggregator_config.py
• tests/models/test_validators.py
• tests/server/services/playbook/test_playbook_aggregator.py
• tests/server/services/playbook/test_playbook_aggregator_clustering.py
• tests/server/services/playbook/test_playbook_generation_service.py
• tests/server/services/storage/test_storage_contract_playbook.py
• tests/server/services/storage/test_storage_contract_retention.py
• tests/server/services/test_embedding_text.py