fix(migrations): baseline adoption for never-migrated DBs + strict Ansible invocation (#10001, #10026)

[mrveiss]

Fixes #10001. Closes the blocking part of #10026 (case 3 + legacy upgrade-path gates); contributes the CI gate requested in #10002.

Thinking Path

The ordering is the mission: both Ansible migration tasks invoked alembic without -c migrations/alembic.ini (guaranteed "No config file" failure) and swallowed it with failed_when: false — so native Postgres deployments never migrated and their schemas (built by historical create_all) carry no alembic_version stamp. 32/37 table-creating migrations use unguarded op.create_table, so making the invocation strict first would strand every such database at migration 001. Therefore: reproduce the stranding (A) → baseline adoption logic (B) → close the create_all faucet (C) → only then strict invocation (D) → permanent gates (E). One commit per phase, every phase proven red-first (details in the report).

What Changed

• python -m migrations.baseline (new): classifies the DB into 4 states (empty / stamped / schema-without-stamp / foreign stamp). Adoption brackets the schema with a self-maintaining probe ladder (per-revision artifacts AST-extracted from the migration files; TIMESTAMPTZ markers for type-only revisions) and stamps the bracketed revision; ambiguous schemas are refused loudly (exit 3/4, nothing written) with a pointer to the new recovery runbook. Never guesses a stamp.
• Faucet closed: migrations/schema_bootstrap.py guard — runtime metadata.create_all allowed only for app-owned SQLite data files unless AUTOBOT_DB_CREATE_ALL=true; static schema-authority sweep keeps the pattern from returning.
• Both playbooks strict (setup-user-backend.yml, update-all-nodes.yml): gate on Postgres-backed mode → pg_dumpall backup → baseline adoption → alembic -c migrations/alembic.ini upgrade head with failed_when removed; update-all-nodes Play 2 gets any_errors_fatal: true and migrations moved after the .env regen / before the restart.
• Permanent gates: migration-gate.yml CI workflow (postgres:16 service) running the (a) empty→head, (b) stamped-intermediate→head, (c) unstamped→baseline→head, (d) head→downgrade-1→head matrix + all static checks; docs/operations/migration-recovery.md operator runbook.

Verification

• 27/27 tests green against disposable Postgres 16 (12 fleet states incl. probe-ladder bracketing of a deliberately old 20260315_010 schema, refusal cases, downgrade round-trip).
• Red-first evidence per phase recorded in MIGRATION_BASELINE_REPORT.md (Phase A committed failing; C guard 4-red; D contract 5-red against old playbooks; sweep red-proven via scratch offender).
• ansible-playbook --syntax-check passes for both playbooks.
• Full suite re-run green after rebase onto latest Dev_new_gui.

Report: MIGRATION_BASELINE_REPORT.md (state machine, ladder contents, fleet states, deliberate non-goals incl. #9980 confirmation and the remote-Postgres backup gap).

Model Used

Claude Fable 5 (claude-fable-5)

🤖 Generated with Claude Code

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

[mrveiss]

Self-review pass (requested follow-up audit) produced one hardening commit and three filed issues:

Fixed on the branch (commit 22b56c955):

• requirements-ci had no alembic → ci.yml's repo-wide pytest on push to Dev_new_gui would have failed the probe-ladder self-checks post-merge. Added alembic==1.18.4 to requirements-ci/storage.txt + pytest.importorskip guard.
• Registered ansible var backend_user_mode shadowed the template variable consumed by backend.env.j2 → renamed to backend_user_mode_probe in both playbooks.

Filed as follow-ups (none blocking this PR):

• bug(migrations): env.py autogenerate view omits models/process_run.py — autogenerate would emit drop_table for live tables #10044 — env.py autogenerate omits models/process_run.py (spurious drop_table risk on revision --autogenerate) + deprecated version_path_separator ini key
• feat(ansible): pre-migration backup must cover remote Postgres — pg_dumpall pattern is local-only #10045 — pre-migration backup is local-pg_dumpall-only; remote-Postgres deploys warn-and-skip, should fail instead
• refactor(ansible): consolidate duplicated backend migration sequence into a shared tasks file #10046 — the strict migration sequence is duplicated across both playbooks; consolidate into a shared tasks file (contract tests are the regression net)

27/27 tests re-verified green after the hardening commit; both playbooks pass ansible-playbook --syntax-check.

[mrveiss]

Review (high effort — 7 finder angles, verified)

Verdict: no blocking findings — approving for merge. Zero confirmed correctness bugs; CI fully green (smoke-test, code-quality, migration-matrix all pass; hardened-smoke-test is the pre-existing #9663 chromadb flake).

Findings (all non-blocking, severity-ranked)

[
  {"file": "autobot-slm-backend/ansible/playbooks/update-all-nodes.yml", "line": 593, "summary": "Mode gate silently skips migrations when .env lacks AUTOBOT_USER_MODE", "failure_scenario": "PLAUSIBLE edge: node misconfigured so .env says single_user while operator intended Postgres → migrations skip. Consistent-by-design: the gate reads the exact file systemd feeds the service (#2824), so a skipped migration implies the runtime is also single_user. Residual risk documented; #10045 will make the backup/migration path fail-closed."},
  {"file": "autobot-slm-backend/ansible/playbooks/update-all-nodes.yml", "line": 428, "summary": "any_errors_fatal: true covers all of Play 2, not just the migration block", "failure_scenario": "A transient non-migration failure (e.g. health check) now aborts the whole rollout. Conscious trade-off per the #10026 mission ('failure on one node must not continue'); documented in the commit. Narrowing to a dedicated migrate play would conflict with the required ordering (after .env regen, before restart)."},
  {"file": "autobot-backend/migrations/schema_bootstrap.py", "line": 27, "summary": "create_all_opted_in() hand-parses an env bool; autobot_shared.env_registry.env() exists", "failure_scenario": "Reuse: registering AUTOBOT_DB_CREATE_ALL in the env registry would consolidate semantics — but couples the dependency-light guard to autobot_shared. Noted on #9841 (env-helper consolidation) rather than blocking."},
  {"file": "autobot-backend/tests/migrations/conftest.py", "line": 76, "summary": "run_alembic/run_baseline duplicate subprocess env setup; script_head() duplicates baseline._script_directory() config", "failure_scenario": "Test-internal duplication (~15 lines). Ride-along candidate for the #10044 PR which touches the same tooling."},
  {"file": "autobot-backend/migrations/db_url.py", "line": 36, "summary": "as_async_url() has 3 string-replace siblings in autobot-slm-backend/migrations", "failure_scenario": "Cross-service divergence risk on URL driver coercion; SLM has its own migration system by design. Consolidation note only."},
  {"file": "autobot-backend/migrations/baseline.py", "line": 289, "summary": "stamp_revision() writes alembic_version via raw SQL instead of alembic API", "failure_scenario": "PLAUSIBLE maintenance cost if alembic ever changes its version-table schema (stable for a decade). In-process alembic stamp would re-enter env.py's async engine path — raw SQL is the deliberate, simpler choice; covered by tests asserting stamp == alembic's own table shape."}
]

Notable refutations (claims checked against code)

• "baseline exit 3/4 isn't caught" — refuted: ansible shell fails on rc≠0 by default; the block aborts and any_errors_fatal stops the rollout.
• "observability allowlist lets new type-only migrations skip silently" — refuted: test_observability_coverage fails on any new unobservable revision, forcing a marker or a conscious allowlist entry.
• "mode gate should use the inventory var instead of grepping .env" — refuted: the .env deployed to the code dir is what systemd feeds the service (fix: backend role should deploy .env from template during provisioning #2824); grepping it reads the runtime's authoritative truth.
• "fresh installs never migrate (role doesn't write .env)" — refuted: roles/backend/tasks/main.yml:619 templates .env into the code dir before post_tasks run.