feat(memory): support-conserving guideline consolidation

[jayaramkr]

What & why

Consolidation currently merges clustered guidelines but discards how many source guidelines back each one and whether they came from successes or failures. That signal is what's needed to dose memory per model — the capability-dependent dosage finding from AppWorld experiments (DeepSeek-V3.2, gpt-oss-120b): strong models absorb the full playbook, weaker models need a small, selective dose. This PR makes consolidation support-conserving so that signal survives.

This is PR 1 of a staged pair. A follow-up adds dosage-aware retrieval (core + top-k selection) that consumes the support signal introduced here.

Changes

• schema (schema/guidelines.py): add support: int (>=1) and evidence: success|failure|both|None to Guideline; add ConsolidatedGuideline/ConsolidatedGuidelineResponse carrying source_indices so consolidation attributes support exactly rather than trusting the model to report counts.
• consolidation (llm/guidelines/clustering.py): combine_cluster now sums member support and merges evidence (success+failure → both) per output guideline via source_indices, and carries any uncovered member through unchanged (lossless fail-safe) so sum(support) is conserved and no advice is ever dropped. New mode arg (lossless default / lossy) selects conservative vs aggressive merging.
• prompt (combine_guidelines.jinja2): asks for 0-based source_indices with lossless framing; adds an aggressive variant for lossy.
• config (config/evolve.py): consolidation_mode (none|lossless|lossy) and lossy_target_num_guidelines. Support-threshold filtering (sup2/sup3) is deliberately left to non-destructive selection in the follow-up, not deletion here.
• client (consolidate_guidelines): passes mode through, short-circuits on none, persists support/evidence, and reports support_before/support_after on ConsolidationResult so conservation is observable.
• generation sites (phoenix_sync, mcp_server): stamp support=1 on freshly mined guidelines.

Tests

Extended tests/unit/test_combine_guidelines.py: support summed, evidence merged, uncovered members carried through (fail-safe), none-mode no-op, and support conserved through consolidate_guidelines. Full unit suite green (395 passed); ruff, ruff format, mypy, and pre-commit all pass. Also verified end-to-end on the filesystem backend (3 originals → 2 consolidated, support 3→3, evidence merged to both).

Compatibility

Backward compatible: support/evidence default to 1/None, ConsolidationResult gains fields with defaults, and consolidate_guidelines(mode=...) is optional (defaults to config). Existing guidelines without the new metadata are treated as support=1, evidence=None.

Summary by CodeRabbit

• New Features
  • Added configurable guideline consolidation modes: none, lossless, and lossy (with a lossy target guideline count).
  • Consolidated guidelines now carry source tracking plus support and evidence; consolidation results report support_before and support_after.
• Bug Fixes
  • Consolidation preserves uncovered guidance instead of dropping it.
  • Support/evidence attribution is now correctly aggregated, including deduping repeated source references.
• Tests
  • Updated and expanded unit tests to cover the new consolidation modes and metadata behavior.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds configurable consolidation modes, support/evidence fields on guidelines, mode-aware clustering with source-index tracking, client wiring for none/lossless/lossy behavior, support metadata at generation sites, and updated tests for support conservation.

Changes

Guideline Consolidation with Support/Evidence Tracking

Layer / File(s)	Summary
Config and schema contracts for support/evidence and consolidation modes altk_evolve/config/evolve.py, altk_evolve/schema/guidelines.py	Adds consolidation mode settings, guideline support/evidence fields, consolidated guideline response types, and support totals on consolidation results.
Mode-driven cluster combination and support attribution altk_evolve/llm/guidelines/clustering.py, altk_evolve/llm/guidelines/prompts/combine_guidelines.jinja2	Updates cluster combination to use mode-aware prompts, consolidated response types, source indices, and support/evidence attribution across merged guidelines.
EvolveClient consolidation wiring altk_evolve/frontend/client/evolve_client.py, altk_evolve/frontend/mcp/mcp_server.py, altk_evolve/sync/phoenix_sync.py	Adds mode handling, none-mode short-circuiting, support and evidence propagation, support totals, and support metadata at guideline creation sites.
Unit tests for consolidation support/evidence behavior tests/unit/test_combine_guidelines.py	Expands fixtures and assertions for support summation, evidence merging, lossy prompt text, uncovered guideline carry-over, and none-mode behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• AgentToolkit/altk-evolve#215: Both PRs change the guideline consolidation pipeline and combine_cluster wiring.
• AgentToolkit/altk-evolve#201: Both PRs modify save_trajectory guideline metadata population in altk_evolve/frontend/mcp/mcp_server.py.

Suggested reviewers: visahak, gaodan-fang, illeatmyhat

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: support-conserving guideline consolidation for memory.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (3)

altk_evolve/llm/guidelines/prompts/combine_guidelines.jinja2 (1)

28-34: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Optional: explicitly forbid repeated indices within one source_indices list.

Given the code-level fix needed in clustering.py's _attribute_support for duplicate indices, consider adding an explicit instruction here (e.g. "Each index must appear at most once within a single guideline's source_indices") as a lightweight extra guardrail, though the primary fix belongs in the parsing logic.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@altk_evolve/llm/guidelines/prompts/combine_guidelines.jinja2` around lines 28
- 34, The consolidation rules need an explicit guardrail against duplicate
indices in a single source_indices list. Update the guidelines in
combine_guidelines.jinja2 to state that each consolidated guideline’s
source_indices must contain only unique 0-based input indices, so no index can
appear more than once within the same list, while still requiring every input
index to appear exactly once across all consolidated guidelines.

altk_evolve/frontend/client/evolve_client.py (1)

227-228: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate support-extraction logic vs. clustering.py.

int((e.metadata or {}).get("support", 1) or 1) is identical to the member_support computation in altk_evolve/llm/guidelines/clustering.py (combine_cluster). Extracting this into a small shared helper (e.g. get_entity_support(entity: RecordedEntity) -> int) would prevent the two call sites from silently drifting and corrupting the support-conservation accounting this PR centers on.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@altk_evolve/frontend/client/evolve_client.py` around lines 227 - 228, The
support extraction for `support_before` in `evolve_client.py` duplicates the
`member_support` logic from `combine_cluster` in `clustering.py`, so move that
metadata-to-int conversion into a shared helper such as
`get_entity_support(entity)` and use it in both places. Update the
`support_before` aggregation to call the helper on each `RecordedEntity`, and
keep `combine_cluster` aligned by using the same helper instead of
reimplementing the fallback logic inline.

tests/unit/test_combine_guidelines.py (1)

49-61: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding a regression test for duplicate indices within one source_indices list.

Given the _attribute_support double-counting bug flagged in clustering.py (duplicate index within a single consolidated guideline's source_indices, e.g. _cg("Merged", "strategy", [0, 0, 1])), a test asserting support isn't inflated by such duplicates would guard against regression once that fix is applied.

Want me to draft this test case?

Also applies to: 91-141

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_combine_guidelines.py` around lines 49 - 61, Add a regression
test around combine_guidelines for duplicate entries inside one consolidated
guideline’s source_indices. Update the test fixture in
test_combine_guidelines.py to include a case like a merged guideline with
repeated indices, then assert the resulting support from the
combine_guidelines/clustering path is not inflated by duplicates. Use the
existing _cg helper and SAMPLE_GUIDELINES setup to locate the test and verify
the _attribute_support behavior stays correct.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@altk_evolve/llm/guidelines/clustering.py`:
- Around line 156-211: The _attribute_support function can double-count support
when a single ConsolidatedGuideline has duplicate entries in source_indices.
Deduplicate the filtered idxs per cg before marking assigned and summing
support, so repeated indices like [0, 0, 1] only contribute once. Keep the same
conservation flow in _attribute_support, including the existing warning if
totals still mismatch, but ensure Guideline.support is computed from unique
member indices only.

In `@altk_evolve/schema/guidelines.py`:
- Around line 90-103: Update the ConsolidationResult docstring in guidelines.py
so it only describes the actual consolidation modes supported by
consolidation_mode in evolve.py; remove the reference to a non-existent
"support" mode and revise the support_before/support_after explanation to match
the current behavior of the consolidation run. Keep the wording aligned with
ConsolidationResult and Consolidation mode handling so the API docs do not imply
a deferred feature is already available.

---

Nitpick comments:
In `@altk_evolve/frontend/client/evolve_client.py`:
- Around line 227-228: The support extraction for `support_before` in
`evolve_client.py` duplicates the `member_support` logic from `combine_cluster`
in `clustering.py`, so move that metadata-to-int conversion into a shared helper
such as `get_entity_support(entity)` and use it in both places. Update the
`support_before` aggregation to call the helper on each `RecordedEntity`, and
keep `combine_cluster` aligned by using the same helper instead of
reimplementing the fallback logic inline.

In `@altk_evolve/llm/guidelines/prompts/combine_guidelines.jinja2`:
- Around line 28-34: The consolidation rules need an explicit guardrail against
duplicate indices in a single source_indices list. Update the guidelines in
combine_guidelines.jinja2 to state that each consolidated guideline’s
source_indices must contain only unique 0-based input indices, so no index can
appear more than once within the same list, while still requiring every input
index to appear exactly once across all consolidated guidelines.

In `@tests/unit/test_combine_guidelines.py`:
- Around line 49-61: Add a regression test around combine_guidelines for
duplicate entries inside one consolidated guideline’s source_indices. Update the
test fixture in test_combine_guidelines.py to include a case like a merged
guideline with repeated indices, then assert the resulting support from the
combine_guidelines/clustering path is not inflated by duplicates. Use the
existing _cg helper and SAMPLE_GUIDELINES setup to locate the test and verify
the _attribute_support behavior stays correct.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e400e1d2-586d-464c-a569-9faf324c470a

📥 Commits

Reviewing files that changed from the base of the PR and between 9d441fd and b886a56.

📒 Files selected for processing (8)

• altk_evolve/config/evolve.py
• altk_evolve/frontend/client/evolve_client.py
• altk_evolve/frontend/mcp/mcp_server.py
• altk_evolve/llm/guidelines/clustering.py
• altk_evolve/llm/guidelines/prompts/combine_guidelines.jinja2
• altk_evolve/schema/guidelines.py
• altk_evolve/sync/phoenix_sync.py
• tests/unit/test_combine_guidelines.py

[visahak]

Summary

This PR adds support-conserving guideline consolidation modes, including lossless, lossy, and none, and threads support metadata through MCP/Phoenix-created guidelines. It also
updates consolidation parsing so LLM outputs attribute merged guidelines back to source indices while preserving total support.

No issues met the confidence threshold for review comments.

• ## Testing

• git diff --check upstream/main...HEAD: passed
• uv run pytest tests/unit/test_combine_guidelines.py -v: 12 passed
• uv run pytest -m e2e -v: ran with .env loaded; 202 passed, 3 skipped, 1 failed, 687 deselected
• Failure details: tests/platform_integrations/test_subscribe.py::TestSubscribe::test_warns_when_audit_write_fails fails in an unrelated platform subscribe path. PR 283 does not
  touch that test, the platform subscribe scripts, or the canonical subscribe script.
• uv run ruff check . --no-fix: passed
• uv run mypy .: passed, no issues found in 124 source files

[visahak]

LGTM.