Skip to content

fix(db): repair migration chain for stale-environment catch-up + replay CI - #97

Open
AlexU-A wants to merge 2 commits into
mainfrom
claude/96-migration-baseline-repair
Open

fix(db): repair migration chain for stale-environment catch-up + replay CI#97
AlexU-A wants to merge 2 commits into
mainfrom
claude/96-migration-baseline-repair

Conversation

@AlexU-A

@AlexU-A AlexU-A commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the v0.1.0 production deploy blocker (#96). Production's __drizzle_migrations log records 0000–0007 from the pre-rewrite chain (db:push era), but its actual schema never received the pipelines objects that the rewritten 0001 creates. The migrator skips by last-applied watermark, so pending work starts at 0008, which ALTERs pipelines objects that don't exist there — crashlooping the boot-time runner.

Reproduced locally before changing anything (seed a scratch DB to journal idx 7, drop the pipelines schema, leave the empty schema triage created, run drizzle-kit migrate): fails with the exact prod error, relation "pipelines.pipelines" does not exist (42P01), failing on 0008's CREATE INDEX. With this PR the same state converges cleanly.

One correction to the issue text: the from-scratch replay path is currently valid on main — verified empirically on the unmodified chain (all 13 migrations apply to an empty database, because the rewritten 0001 creates the pipelines objects). The broken path is specifically stale-environment catch-up. The CI job added here exercises both.

While in here: the journal when timestamps for 0009 and 0012 were non-monotonic (hand-typed), which can silently skip them forever on environments that migrated past them — drizzle-kit applies in journal array order but skips by last-applied created_at watermark (confirmed empirically; a fresh replay records 0009's smaller timestamp after 0008's). Repaired and guarded.

Changes

  • drizzle/migrations/0008_pipeline_name_not_unique.sql — index swap wrapped in a to_regclass('pipelines.pipelines') guard; no-ops where the table doesn't exist yet
  • drizzle/migrations/0013_pipelines_baseline_repair.sql (new) — complete final-state pipelines schema (8 enums, 9 tables, 10 FKs, 30 indexes) with an existence guard on every statement; DDL verbatim from 0001 except the tenant/name index, which is created in its final post-0008 non-unique form. No-op on healthy databases.
  • drizzle/migrations/meta/_journal.json0009 when 1779710400000 → 1779713480600 and 0012 when 1778505114112 → 1779724808298 (restores strict monotonicity); 0013 entry appended
  • drizzle/migrations/0009_tenant_memberships.sql, 0012_jira_a11y_triage_scheduler.sql — idempotency guards (IF NOT EXISTS / DO blocks) so the timestamp repositioning cannot double-apply destructively on environments that recorded them under the old timestamps
  • scripts/migration-replay-check.mjs + npm run db:replay-check (new) — four-phase verifier: journal lint, fresh replay, idempotency, stale-environment catch-up; portable (node + pg only, no psql/docker dependency)
  • .github/workflows/validate.yml — new migration-replay job with a postgres:16-alpine service (matching deploy/docker-compose.yml)
  • CHANGELOG.md — Unreleased entry

Test Plan

  • npm run db:replay-check against local postgres:16-alpine — all 36 checks pass (journal lint ×5, fresh replay ×14, idempotency ×2, prod-shaped catch-up ×15)
  • Reproduced the prod failure on the unmodified chain first with the same harness shape (fails at 0008, 42P01)
  • Manually verified locally
  • Integration tests pass — npm run validate: typecheck, lint, 1601 tests passed (118 files; 50 pre-existing skips)
  • Unit tests added or updated (n/a — verification lives in the replay harness, which CI runs on every PR)

Deploy path for production

No host access needed beyond a normal redeploy of the v0.1.0 image built from this branch/main. Prod's watermark is at 0007, so its pending set becomes 0008 (guarded no-op) → 0009 → 0010 → 0011 → 0012 → 0013 (creates all pipelines objects). The pending batch runs in a single transaction (verified: a failed batch records nothing, so the crashloop left no partial progress), and the empty pipelines schema left by triage is handled by CREATE SCHEMA IF NOT EXISTS.

Suggest keeping #96 open as the deploy tracker and closing it once the image starts cleanly against prod (/health 200 with migrations applied).

Not covered / residual risks

  • The catch-up simulation models the pipelines gap only. If prod diverges from the chain in other ways (e.g. task_type enum missing for 0012), the deploy will surface it with a clear error; the same guarded-repair pattern applies.
  • Long-lived push-managed dev databases (objects present, journal absent or stale) should keep using db:push or reset; 0010/0011 were not repositioned and stay unguarded by design.
  • drizzle-kit check/generate snapshot debt (meta snapshots exist only through 0004) is pre-existing and untouched. The journal-lint phase now pins tag/file/ordering integrity, which is the part that bites at deploy time.

Checklist

  • All tests pass
  • TypeScript / Python types are valid (no new type errors)
  • Documentation updated if behavior changed (CHANGELOG)
  • No secrets, credentials, or client-specific content introduced (scratch-DB URLs are dummy localhost service-container credentials)
  • Follows the Client Abstraction rule (§2.10): no real names, client names, or domain-specific jargon
  • PR title is descriptive and follows conventional commit style if applicable
  • I classified this PR correctly:
    • Idea lane only (ideas/**), no governed spec changes
    • Governed spec change (spec/**, kitty-specs/**, .claude/commands/**, .kittify/**, README.md, or ROADMAP.md)
    • Neither lane applies: runtime migration + CI change only (no governed paths touched)

Refs #96

🤖 Generated with Claude Code

AlexU-A and others added 2 commits June 10, 2026 15:32
The production database recorded migrations 0000-0007 against the
pre-rewrite chain (db:push era), so it never received the pipelines
objects that the rewritten 0001 creates. The migrator's watermark skips
0001 there, and 0008+ then ALTER pipelines objects that do not exist,
crashlooping the boot-time runner (relation "pipelines.pipelines" does
not exist, 42P01). Reproduced locally by seeding a scratch database to
journal idx 7 and dropping the pipelines schema.

- guard 0008 so the index swap no-ops where pipelines.pipelines is absent
- add 0013_pipelines_baseline_repair: complete final-state pipelines
  schema (enums, tables, FKs, indexes) with existence guards on every
  statement; verbatim 0001 DDL except the tenant/name index, created in
  its final post-0008 non-unique form. No-op on healthy databases.
- repair journal when monotonicity: 0009 (1779710400000 -> 1779713480600)
  and 0012 (1778505114112 -> 1779724808298) sorted before earlier entries
  and could be silently skipped by environments that migrated past them
  (drizzle-kit applies in journal order but skips by last-applied
  created_at watermark)
- make 0009/0012 idempotent so the reposition cannot double-apply
  destructively on environments that recorded them under old timestamps

Refs #96

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add npm run db:replay-check (scripts/migration-replay-check.mjs) and a
migration-replay CI job (postgres:16-alpine service, matching prod) so
replay-path breakage like #96 cannot hide again:

1. journal lint: idx contiguity, strictly increasing when, unique tags,
   tag <-> file parity in both directions
2. fresh replay: full chain against an empty scratch database, then
   assert the converged pipelines state
3. idempotency: a second migrate run applies nothing
4. stale-environment catch-up: seed to journal idx 7, drop the pipelines
   schema (leaving the empty schema production triage left behind), run
   the full chain, assert convergence

All 36 checks pass locally against postgres:16-alpine.

Refs #96

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants