fix(lineage): dedup path always soft-supersedes + commit-atomic legacy log

[yilu331]

Summary

Closes the dedup-path hard-delete instrumentation gap — the last write-side prerequisite for retiring the legacy ProfileChangeLog (Track-A). The profile deduplication path now always soft-supersedes removed profiles instead of sometimes hard-deleting them unreconstructably, and the legacy change-log is made commit-atomic so it can never record a removal that didn't actually commit.

The problem

The dedup removal path had a hard-delete fallback (when is_dedup_soft_delete_enabled was off or request_id was empty) that DELETEd the row and emitted a hard_delete event under a throwaway uuid4() — uncorrelatable to the run, so reconstruction could never reproduce it. Separately, the legacy ProfileChangeLog.removed_profiles was built from the deduplicator's intent, written unconditionally even if supersede threw/skipped/partially no-op'd → a phantom "removed" entry with no matching tombstone → permanent CONTENT_MISMATCH blocking the parity gate.

The fix

• Always soft-supersede dedup removals (removed the hard-delete fallback and the now-dead is_dedup_soft_delete_enabled flag).
• supersede_profiles_by_ids now returns the committed ids (list[str], was int rowcount) across all backends, so the service knows exactly what was tombstoned.
• Commit-atomic legacy log: removed_profiles is built from the committed set, not the intent. Supersede exception → committed=[]; empty request_id → fail-loud capture_anomaly("lineage.dedup.missing_request_id") + skip (never hard-delete). Legacy and reconstruction agree by construction.
• The two legitimate hard-delete callers (user-initiated delete, admin org-wipe) are untouched.

Invariant + tests

A dedup removal either soft-supersedes (tombstone + one status_change/superseded event under the run's request_id) AND appears in legacy removed_profiles — or neither; never a phantom removal, never a hard_delete. New test_dedup_always_soft_integration.py: real-storage happy path asserting reconstruction MATCH, plus supersede-raises / partial-commit / empty-request_id / regression guards.

Review

Plan reviewed via /review-design-doc (3 lenses; the adversarial lens surfaced the phantom-removal/atomicity class this fix kills). Built via subagent-driven-development with per-task reviews + a whole-branch (opus) review that traced the invariant across all 3 backends.

Plan: docs/superpowers/plans/2026-06-22-lineage-dedup-hard-delete-fix.md (enterprise repo).

Summary by CodeRabbit

Release Notes

• New Features

  • Deduplication removals are now always soft-deleted, permanently preserving profiles as superseded records.

• Bug Fixes

  • Profile removal tracking now prevents phantom removals.
  • Missing request IDs are now properly detected and anomalies are captured instead of causing silent failures.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b58414ee-13f9-430d-8e6a-13305c98f3d5

📥 Commits

Reviewing files that changed from the base of the PR and between 6e143b1 and 79a07ec.

📒 Files selected for processing (9)

• reflexio/server/services/profile/profile_generation_service.py
• reflexio/server/services/storage/sqlite_storage/_profiles.py
• reflexio/server/services/storage/storage_base/_profiles.py
• reflexio/server/site_var/feature_flags.py
• tests/server/services/profile/test_dedup_always_soft_integration.py
• tests/server/services/profile/test_dedup_soft_delete_integration.py
• tests/server/services/storage/test_lineage_b2gc_retired_at_integration.py
• tests/server/services/storage/test_lineage_b3c_profile_integration.py
• tests/server/site_var/test_feature_flags.py

💤 Files with no reviewable changes (2)

• tests/server/services/profile/test_dedup_soft_delete_integration.py
• reflexio/server/site_var/feature_flags.py

---

📝 Walkthrough

Walkthrough

Dedup removals are made unconditionally soft-superseded by changing supersede_profiles_by_ids to return a list[str] of committed profile IDs instead of an integer count. The service's _finalize_extracted_items removes the feature-flag-gated hard-delete branch, adds an anomaly capture for missing request_id, and derives legacy changelog "removed" entries only from IDs that actually committed. The is_dedup_soft_delete_enabled flag is deleted entirely.

Changes

Always-soft dedup supersede

Layer / File(s)	Summary
supersede_profiles_by_ids: int → list[str] contract reflexio/server/services/storage/storage_base/_profiles.py, reflexio/server/services/storage/sqlite_storage/_profiles.py	Abstract base return type and docstring updated to list[str]; SQLite implementation replaces the updated integer counter with a committed_ids list accumulator, returning it after commit. Early-return for empty profile_ids now returns [].
Service finalization: always-soft path, committed-ID tracking, anomaly guard, changelog fix reflexio/server/services/profile/profile_generation_service.py	Removes DeleteUserProfileRequest import; adds capture_anomaly. Replaces feature-flagged hard-delete/supersede branching with an unconditional supersede_profiles_by_ids call that captures committed_ids. When request_id is empty, skips removal and fires capture_anomaly("lineage.dedup.missing_request_id"). removed_for_log is computed as the intersection of superseded_profiles and committed_ids; changelog creation is gated on real committed removals.
Remove is_dedup_soft_delete_enabled and its tests reflexio/server/site_var/feature_flags.py, tests/server/site_var/test_feature_flags.py	Deletes the is_dedup_soft_delete_enabled helper. Removes the corresponding import, TestDedupSoftDeleteFlag class, and dedup_soft_delete-specific malformed-config and strict-bool test cases from the test suite; aggregation_soft_delete coverage is retained.
New always-soft integration tests + updated storage tests tests/server/services/profile/test_dedup_always_soft_integration.py, tests/server/services/storage/test_lineage_b2gc_retired_at_integration.py, tests/server/services/storage/test_lineage_b3c_profile_integration.py	Adds a four-case integration suite: SQLite happy path (tombstone status, lineage event, no hard-delete, changelog parity), failure/partial-commit with mocked storage, empty request_id anomaly path, and regression guard against hard-delete. Removes the old test_dedup_soft_delete_integration.py. Updates two existing storage tests to assert list return values instead of integer counts.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• ReflexioAI/reflexio#191: Directly precedes this PR — introduced the feature-flag-driven supersede/hard-delete branching in _finalize_extracted_items and the original supersede_profiles_by_ids integer return that this PR replaces.
• ReflexioAI/reflexio#196: Added the storage-layer request_id non-empty guard in supersede_profiles_by_ids; this PR lifts that guard into the service layer as an anomaly-capture path.
• ReflexioAI/reflexio#201: Overlaps directly on feature_flags.py for dedup_soft_delete default-on evaluation, which this PR removes entirely.

Poem

🐇 No more flags to flip, no hard-delete to dread,
The tombstone path is carved — soft-supersede instead!
Committed IDs returned, a list crisp and true,
Anomaly captured when request_id flew.
The changelog now reflects only what's really gone —
Hop hop, invariant held, the dedup marches on! 🪦✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly describes the main change: dedup path now always soft-supersedes and makes legacy log commit-atomic instead of conditional.
Docstring Coverage	✅ Passed	Docstring coverage is 81.25% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/lineage-dedup-always-soft

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}