ci(deploy): the code-sync post-sync step order is correct but unpinned — a reordering that restarts before migrating would pass CI #17867

The gap

Every builtin update funnels through one of four branch functions in autobot-slm-backend/api/code_sync.py. Each hard-codes its sequence imperatively. Verified by AST extraction, all four are correct today:

_run_post_sync_backend_branch
    constraints -> repo-root requirements -> ensure python -> ensure venv
    -> pip install -> reconcile (removals) -> alembic -> shared symlink
    -> restart -> health
_run_post_sync_frontend_branch   npm ci+build -> restart -> health
_run_post_sync_worker_branch     pip install -> reconcile -> restart -> health
_run_post_sync_shared_branch     shared symlink -> restart dependents

Rollback is wired on pip failure, alembic failure and unhealthy-after-restart, and autobot_shared-first ordering is enforced separately with its own deletion guard (#11611, #13851).

Nothing holds that order in place.

Why the existing protection does not reach it

The sibling path is protected. roles.py builds its action list through ordered_post_sync_plan, and test_post_sync_plan_order.py pins install -> schema -> build -> restart, asserts no unit restarts twice, and carries a negative control.

code_sync.py never calls ordered_post_sync_plan — zero references. It is a different mechanism, so that contract does not apply to it. The only ordering fact pinned on this path was "a failed pip install short-circuits before reconcile" (test_venv_reconcile_wiring_15063.py), which covers the first link and none of the rest.

So each of these would pass CI today:

• restart moved ahead of the alembic migration — new code against an un-migrated schema
• reconcile moved ahead of the pip install — removals computed against a set the venv has not been brought to
• _ensure_autobot_shared_symlink moved after the restart — the exact crash-loop CRITICAL: deploying a component before autobot_shared crash-loops it — SLM control plane went down (missing autobot_shared.db_url) #11611 exists to prevent, which once took the control plane offline and needed manual recovery
• npm run build moved after the nginx reload — serving a half-built asset tree

This is the shape where correct and protected get conflated: reading the code shows a sound sequence, and reading CI shows green, and neither says the sequence is held.

Acceptance criteria

• A guard pins the relative step order of all four branches, and fails when any pair is swapped. Prove it by reordering the real file and showing a red, not by reading the test.
• A single invariant covers steps the guard does not yet know about: no mutating step may run after the first restart in its branch. Per-pair ordering alone cannot catch a newly appended step.
• Health-after-restart is pinned wherever both appear, since health is what gates rollback.
• A renamed or deleted branch function fails the guard rather than silently dropping out of coverage.
• Negative controls, so none of the above can pass vacuously.

One trap for whoever implements this

ast.walk is breadth-first, not source order. Extracting awaited calls with it reports reconcile_component running after the rollback in the worker branch, and the shared branch restarting before its symlink restore. Both look exactly like real sequencing bugs and both are artefacts of the traversal.

I hit this while writing the extraction and briefly believed the shared branch had a genuine ordering defect. A guard built on walk order would fail on correct code and then be "fixed" by reordering the source — making the deploy path worse to satisfy a broken check. Sort by lineno.