feat(migration): Session & Secret Data Migration (#875)

[mrveiss]

Summary

Implements #875 - Migration scripts for session and secret data.

Part of #608 - User-Centric Session Tracking (Phase 7)

Changes

Migration Scripts Created

1. migrate_sessions_user_attribution.py

   • Adds owner_id to 74 existing sessions
   • Infers owner from first message sender
   • Adds created_at timestamps if missing
   • Registers ownership in Redis indices (user/org/team)
   • Generates rollback SQL

2. migrate_secrets_ownership.py

   • Adds owner_id and scope to secrets
   • Infers owner from creation metadata or session
   • Ensures all secret values are encrypted
   • Registers ownership in Redis indices

3. cleanup_orphaned_sessions.py

   • Identifies and removes 45 empty/orphaned sessions
   • Sessions with no messages AND no activities
   • Configurable minimum age (default: 7 days)
   • Lists all deleted sessions

4. backfill_activity_user_ids.py

   • Adds user_id to existing chat messages
   • Links activities to users based on session owner
   • Batch processing for performance

5. validate_migration.py

   • Comprehensive validation of migration success
   • Verifies all sessions have owner_id
   • Verifies all secrets have owner_id and scope
   • Checks encryption status
   • Generates detailed validation report
   • Exit code 1 if errors found (for CI/CD)

Features

All scripts include:

• --dry-run mode - Preview changes without applying
• Detailed logging - All operations logged with timestamps
• Rollback generation - SQL/notes for reverting changes
• Idempotency - Safe to run multiple times
• Error handling - Graceful handling of individual failures
• Statistics reporting - Summary of all operations

Documentation

• Comprehensive README.md with:
  • Usage examples for each script
  • Recommended 3-phase migration workflow
  • Expected data volumes (74 sessions, 45 to delete)
  • Troubleshooting guide
  • Rollback procedures

Testing

Dry-Run Tests

# All scripts tested in dry-run mode
python3 migrate_sessions_user_attribution.py --dry-run
python3 migrate_secrets_ownership.py --dry-run
python3 backfill_activity_user_ids.py --dry-run
python3 cleanup_orphaned_sessions.py --dry-run
python3 validate_migration.py

Code Quality

• ✅ Flake8 passes (max-line-length=100)
• ⚠️ Function length warnings (migration scripts, one-time use)
• ✅ All scripts executable
• ✅ Proper logging (no print statements except in report output)

Production Execution Plan

Prerequisites:

1. Backup production database: redis-cli BGSAVE
2. Review dry-run output for all scripts
3. Schedule during low-traffic window

Execution Order:

1. migrate_sessions_user_attribution.py (add owners to 74 sessions)
2. migrate_secrets_ownership.py (add owners to secrets)
3. backfill_activity_user_ids.py (link messages/activities to users)
4. validate_migration.py (verify success)
5. cleanup_orphaned_sessions.py (remove 45 empty sessions)
6. Final validate_migration.py --verbose

Expected Results:

• 74 sessions migrated with owner_id
• All secrets have owner_id and scope
• 45 orphaned sessions removed
• All messages have user_id attribution
• Redis indices populated correctly

Rollback Plan

Each script generates rollback documentation:

• /tmp/session_migration_rollback.sql
• /tmp/secrets_migration_rollback.sql
• /tmp/deleted_sessions.txt

Manual reversal required (Redis operations documented in rollback files).

Acceptance Criteria

• All 5 migration scripts created
• Scripts support --dry-run mode
• Detailed logging implemented
• Rollback generation included
• Validation script with exit codes
• Comprehensive README with workflow
• Code quality checks pass (flake8)
• Production execution (pending approval)
• Validation report confirms 100% migration

Dependencies

None - scripts are standalone and can run independently.

Related Issues

• Parent: [Feature] User-Centric Session Tracking for All UI Components #608 - User-Centric Session Tracking
• Phase 7 of 7-phase implementation

Closes #875

🤖 Generated with Claude Code