fix(db): repair migration chain for stale-environment catch-up + replay CI - #97
Open
AlexU-A wants to merge 2 commits into
Open
fix(db): repair migration chain for stale-environment catch-up + replay CI#97AlexU-A wants to merge 2 commits into
AlexU-A wants to merge 2 commits into
Conversation
The production database recorded migrations 0000-0007 against the pre-rewrite chain (db:push era), so it never received the pipelines objects that the rewritten 0001 creates. The migrator's watermark skips 0001 there, and 0008+ then ALTER pipelines objects that do not exist, crashlooping the boot-time runner (relation "pipelines.pipelines" does not exist, 42P01). Reproduced locally by seeding a scratch database to journal idx 7 and dropping the pipelines schema. - guard 0008 so the index swap no-ops where pipelines.pipelines is absent - add 0013_pipelines_baseline_repair: complete final-state pipelines schema (enums, tables, FKs, indexes) with existence guards on every statement; verbatim 0001 DDL except the tenant/name index, created in its final post-0008 non-unique form. No-op on healthy databases. - repair journal when monotonicity: 0009 (1779710400000 -> 1779713480600) and 0012 (1778505114112 -> 1779724808298) sorted before earlier entries and could be silently skipped by environments that migrated past them (drizzle-kit applies in journal order but skips by last-applied created_at watermark) - make 0009/0012 idempotent so the reposition cannot double-apply destructively on environments that recorded them under old timestamps Refs #96 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add npm run db:replay-check (scripts/migration-replay-check.mjs) and a migration-replay CI job (postgres:16-alpine service, matching prod) so replay-path breakage like #96 cannot hide again: 1. journal lint: idx contiguity, strictly increasing when, unique tags, tag <-> file parity in both directions 2. fresh replay: full chain against an empty scratch database, then assert the converged pipelines state 3. idempotency: a second migrate run applies nothing 4. stale-environment catch-up: seed to journal idx 7, drop the pipelines schema (leaving the empty schema production triage left behind), run the full chain, assert convergence All 36 checks pass locally against postgres:16-alpine. Refs #96 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the v0.1.0 production deploy blocker (#96). Production's
__drizzle_migrationslog records 0000–0007 from the pre-rewrite chain (db:push era), but its actual schema never received thepipelinesobjects that the rewritten0001creates. The migrator skips by last-applied watermark, so pending work starts at0008, which ALTERs pipelines objects that don't exist there — crashlooping the boot-time runner.Reproduced locally before changing anything (seed a scratch DB to journal idx 7, drop the pipelines schema, leave the empty schema triage created, run
drizzle-kit migrate): fails with the exact prod error,relation "pipelines.pipelines" does not exist(42P01), failing on0008'sCREATE INDEX. With this PR the same state converges cleanly.One correction to the issue text: the from-scratch replay path is currently valid on
main— verified empirically on the unmodified chain (all 13 migrations apply to an empty database, because the rewritten0001creates the pipelines objects). The broken path is specifically stale-environment catch-up. The CI job added here exercises both.While in here: the journal
whentimestamps for0009and0012were non-monotonic (hand-typed), which can silently skip them forever on environments that migrated past them — drizzle-kit applies in journal array order but skips by last-appliedcreated_atwatermark (confirmed empirically; a fresh replay records0009's smaller timestamp after0008's). Repaired and guarded.Changes
drizzle/migrations/0008_pipeline_name_not_unique.sql— index swap wrapped in ato_regclass('pipelines.pipelines')guard; no-ops where the table doesn't exist yetdrizzle/migrations/0013_pipelines_baseline_repair.sql(new) — complete final-statepipelinesschema (8 enums, 9 tables, 10 FKs, 30 indexes) with an existence guard on every statement; DDL verbatim from0001except the tenant/name index, which is created in its final post-0008non-unique form. No-op on healthy databases.drizzle/migrations/meta/_journal.json—0009when1779710400000 → 1779713480600 and0012when1778505114112 → 1779724808298 (restores strict monotonicity);0013entry appendeddrizzle/migrations/0009_tenant_memberships.sql,0012_jira_a11y_triage_scheduler.sql— idempotency guards (IF NOT EXISTS/DOblocks) so the timestamp repositioning cannot double-apply destructively on environments that recorded them under the old timestampsscripts/migration-replay-check.mjs+npm run db:replay-check(new) — four-phase verifier: journal lint, fresh replay, idempotency, stale-environment catch-up; portable (node +pgonly, no psql/docker dependency).github/workflows/validate.yml— newmigration-replayjob with apostgres:16-alpineservice (matchingdeploy/docker-compose.yml)CHANGELOG.md— Unreleased entryTest Plan
npm run db:replay-checkagainst local postgres:16-alpine — all 36 checks pass (journal lint ×5, fresh replay ×14, idempotency ×2, prod-shaped catch-up ×15)0008, 42P01)npm run validate: typecheck, lint, 1601 tests passed (118 files; 50 pre-existing skips)Deploy path for production
No host access needed beyond a normal redeploy of the v0.1.0 image built from this branch/main. Prod's watermark is at
0007, so its pending set becomes0008 (guarded no-op) → 0009 → 0010 → 0011 → 0012 → 0013 (creates all pipelines objects). The pending batch runs in a single transaction (verified: a failed batch records nothing, so the crashloop left no partial progress), and the emptypipelinesschema left by triage is handled byCREATE SCHEMA IF NOT EXISTS.Suggest keeping #96 open as the deploy tracker and closing it once the image starts cleanly against prod (
/health200 with migrations applied).Not covered / residual risks
task_typeenum missing for0012), the deploy will surface it with a clear error; the same guarded-repair pattern applies.db:pushor reset;0010/0011were not repositioned and stay unguarded by design.check/generatesnapshot debt (meta snapshots exist only through0004) is pre-existing and untouched. The journal-lint phase now pins tag/file/ordering integrity, which is the part that bites at deploy time.Checklist
ideas/**), no governed spec changesspec/**,kitty-specs/**,.claude/commands/**,.kittify/**,README.md, orROADMAP.md)Refs #96
🤖 Generated with Claude Code