Run Library migrations on the deployment candidate - #193
Conversation
Tachyon Preview
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c04e2a992d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else { | ||
| return false; | ||
| }; | ||
| constant_time_compare(provided.as_bytes(), expected.as_bytes()) |
There was a problem hiding this comment.
When LIBRARY_DEPLOY_HOOK_TOKEN is present but empty, a request with Authorization: Bearer reaches this comparison with both byte slices empty, so the public deploy-migration endpoint authorizes and runs production migrations with a trivially guessable token. Treat an empty expected token the same as missing configuration before comparing.
Useful? React with 👍 / 👎.
| .route( | ||
| "/internal/deploy/migrate", | ||
| post(handler::deploy::run_migrations), | ||
| ) |
There was a problem hiding this comment.
In a fresh or recovered production database where library or tachyon_apps_database_manager does not exist yet, this hook can never reach run_migrations: the normal Lambda entrypoint constructs database-manager state and router() opens pools against those schemas before this route is available, while the migration runner is the code that would create them. The previous dedicated migration Lambda could connect to the admin DSN first, but invoking the fully booted candidate API deadlocks first-time/schema-recovery deploys on the missing database.
Useful? React with 👍 / 👎.
| run: >- | ||
| set -eu; | ||
| wget --quiet --output-document=/dev/null | ||
| --header="Authorization: Bearer $LIBRARY_DEPLOY_HOOK_TOKEN" |
There was a problem hiding this comment.
In the production hook, the shell expands $LIBRARY_DEPLOY_HOOK_TOKEN into wget's --header argument, so every hook run places the token that authorizes the public migration endpoint in the process command line, where hook-runner diagnostics or process-list capture can expose it. Pass the header through a protected temporary config/file descriptor instead of argv so the secret stays out of command metadata.
Useful? React with 👍 / 👎.
1 participant
Summary
lambda-library-api-migratepre-deploy invocation with a candidate API migration gatePOST /internal/deploy/migrateendpoint and call it from the production candidate lifecycle hook before Alias promotionlibrary-apifrom1.11.2onorigin/mainto1.11.3(patch)Root cause
The dedicated migration Lambda reused a warm execution environment and attempted to initialize the global tracing subscriber again. Its second invocation panicked, which prevented the otherwise successful Library API build from being promoted to production.
The generic command pre-deploy JobRun has neither repository checkout nor the API build artifact, so it cannot run the Rust/SQLx migration directly. The candidate API carries the exact migration set for its build and is available before production Alias promotion, making the lifecycle hook a pre-release gate.
Validation
cargo +nightly-2026-06-04 fmt --all -- --checkcargo +nightly-2026-06-04 test -p library-api --lib --no-default-features— 105 passed, 3 ignoredcargo +nightly-2026-06-04 clippy -p library-api --no-default-features --all-targets -- -D warningscargo +nightly-2026-06-04 build -p library-api --bin lambda-library-api --no-default-featuresProduction follow-up after merge
/versionreports1.11.3Production verification is intentionally pending because this commit has not yet been merged into the production branch.
Docs
docs/specs/decisions/ADR-0007-library-candidate-migration-gate.mddocs/src/tasks/completed/v1.11.3/library-candidate-migration-hook/verification-report.md