Incarnation becomes unrecoverable when change history contains invalid commit SHAs

[peterbraden]

Problem

An incarnation can enter a deadlock state where:

GET /api/incarnations/{id} returns an error: "the given change is in an incomplete state (commit_pushed=False). Try POST .../fix"
Calling the suggested fix endpoint returns {"detail": "Change not found"} (404)

All mutation endpoints (PUT, PATCH, POST .../changes) refuse to operate because they require the latest change to be complete
There is no API path to recover — the only option is direct database manipulation

Root causes

Two scenarios can produce this state:

1. Force-push or revert rewrites repository history

When someone force-pushes to the incarnation repository or reverts commits, the commit_sha values stored in foxops' change table become invalid — those commits no longer exist in the repo. If multiple revisions are affected:

The fix endpoint deletes the latest broken change (revision N), but the previous revision (N-1) is also broken.

Revision N-1 may have commit_pushed=True (it was completed before the force-push), so the fix endpoint considers it "already complete" and does nothing — even though its SHA points at a nonexistent commit.

The incarnation is now stuck: GET reports an error, fix can't help.

2. Stale foxops update branch not cleaned up

The branch name for MR changes is deterministic: foxops/update-to-{target_dir_hash}-{version}. When an MR is merged, GitLab deletes the source branch (via remove_source_branch), but if the MR is closed without merging, or merged outside GitLab's MR flow, the branch persists.

When a subsequent update targets the same version (e.g. with different template data), _prepared_change_environment generates the same branch name. The local git checkout -b succeeds (the GitLab clone is --depth=1 and doesn't see the remote branch), and the DB row is created with commit_pushed=False. But git push fails because the branch already exists on the remote with different history, triggering RebaseRequiredError.

The retry loop in _push_change_commit_and_update_database does git pull --rebase, which rebases the new commit onto the stale branch's history. This can:

• Drop the commit entirely (if the rebase determines the changes are already applied), leaving the DB with a SHA that doesn't exist
• Produce conflicts and exhaust all 10 retries, at which point the change is deleted — but if the process crashes or times out before cleanup, the orphaned commit_pushed=False row remains
  Succeed but with a different SHA that gets recorded via update_commit_sha, creating a mismatch

In all cases, the result is the same unrecoverable state.

[peterbraden]

Analysis findings

The deadlock described in this issue stems from an architectural mismatch: the change table serves simultaneously as a request log (what foxops was asked to do) and as the source of truth for incarnation state (what commit the incarnation is currently at). Git — the actual source of truth — is only consulted during active operations.

When these diverge (force-push, stale branches), the current repair mechanism (update_incomplete_change) deletes change records to reconcile with git. This is problematic because:

• It destroys audit history (what was requested, when, by whom)
• It only fixes the topmost change, leaving deeper broken records to surface one at a time
• The commit_pushed=True early-return means changes with valid flags but nonexistent commits are treated as healthy and never repaired

Proposed fix direction

Rather than deleting changes, introduce an invalidation model:

• Add a validity/status column to the change table (e.g. valid, invalidated)
• "Latest change" queries skip invalidated records
• The fix operation marks changes as invalidated rather than deleting them, preserving the request history
• Validation always checks whether the commit SHA exists in git, regardless of commit_pushed flag
• Replace the per-revision {revision}/fix endpoint with an incarnation-level repair that walks the chain until a valid change is found

Additionally, prevent the stale-branch scenario by checking for (and cleaning up) orphaned remote branches before creating new ones in _prepared_change_environment.