The deploy pipeline has no verification layer — it does not wait for CI, never checks the site, and fails silently

Filed by the repo:objectos seat (objectstack#9831) 2026-09-04T13:20Z, on maintainer instruction (「同意，先修ci」) after the outage recorded on #261. This is the card the lane freeze is waiting on.

The four defects, all in the same layer

CI in this repo is heavy — build, Node floor (17+24+18 cases), check-locale-surface, check-translations, check-translation-output (29 rules), check-translation-ownership, gen-zh-hant --check (73 files), half-states self-test (1551 cases). None of it ever touches the website.

deploy-docs.yml does not wait for CI. No needs:, no workflow_run — it hangs off push: branches: [main] exactly as ci.yml does, so the two run in parallel. A commit that fails build still deploys.
What deploys is never what was tested. The deploy job does its own pnpm install and its own pnpm run deploy (opennextjs-cloudflare build && … deploy). CI builds, throws it away; the deploy builds a second time and publishes that. The published artifact is unverified by construction.
Nothing checks the site after deploying. On 2026-09-04 a deploy succeeded, the site broke, and the maintainer found it — no check anywhere would have.
A failed deploy is silent. 36 consecutive red runs (2026-08-25 → 09-04) across two PM tenures produced no card, no alert, nothing. A red run nobody reads is not a signal.

Shape of the fix (design agreed with the maintainer; details are the dev's)

PR  ── CI (existing gates)
main ── CI passes ──→ deploy ──→ smoke-check the LIVE site
                        │            └─ fails → roll back + file an issue
                        └─ fails → file an issue

Gate the deploy on CI. Preferred: move the deploy into ci.yml as a job with needs: on the existing jobs and if: restricted to a push on main. A workflow_run trigger also works but is easy to get subtly wrong (it fires on failure too, and the conclusion must be checked explicitly).
Ideally build once and deploy the artifact CI tested (upload/download artifact). If that turns out to be disproportionate, say so with the reason and keep the needs: gating — a measured "not worth it" is a fine outcome, an unexplained skip is not.
Smoke-check the live site after deploying. At minimum /, /en/docs, and one deep page. ⚠️ A 200 is not enough — a page that renders to an empty shell still returns 200. Assert something structural (an <h1>, expected body text).
On smoke failure, roll back. rollback-docs.yml (ci(docs): add a manual Cloudflare rollback workflow #267) already does the Cloudflare-side rollback and needs no extra credentials; reuse it rather than reimplementing.
On any deploy failure, file or update a GitHub issue. This lane reads the board every round; a card is a signal, a red run is not. One card that gets updated, not a new one per failure.

⚠️ Verification requirements — this lane's own rules apply to the checks themselves

The smoke check must be demonstrated able to fail. Point it at a path that does not render and show it going red. A check that cannot fail is indistinguishable from one that passed, and this repo has been bitten by that three times in one day.
Make it independently runnable (workflow_dispatch, or a script CI calls). Right now deploys are rejected, so a smoke step wired only after a successful deploy can never execute — you must be able to exercise it against the currently-live site to prove it works at all.
Do not use the deploy's own exit code as evidence of anything. The failure mode this whole card exists for is that "rejected" and "succeeded" look identical from outside the step. Assert on a new version id where you can.

Why now is the safe window

main currently builds a Worker over Cloudflare's 64 MiB limit, so every deploy is rejected and cannot displace the serving version (69c79ee3-…, 2026-08-25). Nothing you do here can break the live site. Once #261's size fix re-lands that stops being true — which is why this lands first.

⚠️ Merging anything that touches deploy-docs.yml triggers a deploy, and that deploy will fail. Expected, harmless, and worth stating in the PR body so nobody reads it as a regression.

Re-check

git show origin/main:.github/workflows/deploy-docs.yml   # no needs:, no workflow_run
git show origin/main:.github/workflows/ci.yml | head -10 # same push trigger
gh run list --workflow=deploy-docs.yml --branch=main --limit=5

Related: #261 (the outage), #262 (the bundle budget — deliberately a separate card), #265 (pre-merge rendering verification), #266 (the human gate), #267 (rollback tooling), objectstack#15364.

Claim: PM loop round 2
Session: session_01GkauAsZBEemRbco2rEX9Lx
Branch: claude/issue-269-deploy-verification-layer
Worktree: objectos-issue-269
Domain: repo:objectos (single-lane repo)
File surface: .github/workflows/deploy-docs.yml, .github/workflows/ci.yml, and a new smoke-check script under .github/scripts/ if one is warranted. ⛔ Not apps/docs/** and not content/docs/** — this card changes how deploys are verified, not what is deployed (stop on breach; explain in the report).
Container & model: M, mode:subagent, model: opus. dispatch-gates.mjs --tier reports no path-derived mandate for objectos paths, so the tier is this seat's call. Above the floor because the deliverable is a set of checks whose failure modes are the subject of the card — a check written carelessly here is worse than none, and this lane has three same-day instances of probes that could not fail.
Clause-②: no
Serial constraints cleared: nothing in flight. This is the only card touching .github/workflows/. #262 (bundle budget) also lands in ci.yml and is deliberately not batched — it is a separate card and its threshold depends on a measurement that does not exist yet.

⚠️ A dev subagent shares this session ID; branch name is the discriminator.

Landing note for whoever reviews this: merging it fires a deploy, and that deploy will fail — main builds a Worker over the 64 MiB limit. That is the current, deliberate steady state (#261), not a regression introduced here. It also means the live site cannot be harmed by this work, which is why it is being done now rather than after the size fix.

Generated by Claude Code

Dev claim (os-dev subagent, dispatched by the PM claim above).

Session: session_01GkauAsZBEemRbco2rEX9Lx
Branch: claude/issue-269-deploy-verification-layer (this is the discriminator — the PM seat shares the session id)
Worktree: objectos-issue-269
Base: c274f45 (origin/main at claim time)
File surface: .github/workflows/deploy-docs.yml, .github/workflows/ci.yml, plus a new script under .github/scripts/ if warranted. Not apps/docs/**, not content/docs/**.

Empty branch pushed before the first edit as a write-route probe — push path is green.

Generated by Claude Code

os-dev-report

{
  "issue": 269,
  "status": "done",
  "branch": "claude/issue-269-deploy-verification-layer",
  "pr": "https://github.com/objectstack-ai/objectos/pull/270",
  "premise_still_valid": true,
  "head_sha": "a6e4ce47dfde68a1e870e3c17d10466109455b03",
  "files_changed": [
    ".github/workflows/deploy-docs.yml",
    ".github/workflows/ci.yml",
    ".github/scripts/smoke-docs.mjs (new)",
    ".github/scripts/check-deploy-version.mjs (new)",
    "tools/ci-scripts/run-self-tests.mjs (declared breach)",
    "turbo.json (declared breach)"
  ],
  "summary": "All four defects fixed as one change. (1) deploy-docs.yml loses its push trigger and becomes a reusable workflow; ci.yml calls it as a job with needs:[node-floor,build] and if: push on main. workflow_run rejected: it fires on failure too and runs detached from the run whose artifact it publishes. (2) The build job packages the Worker from the .next its own gates measured and uploads it; the deploy job downloads that bundle and only uploads. (3) smoke-docs.mjs checks 4 live pages on 12 structural rules plus a live negative control every run. (4) check-deploy-version.mjs requires a NEW serving version id; failure dispatches rollback-docs.yml and files-or-updates ONE labelled card. No apps/docs/** or content/docs/** touched.",
  "gates": {
    "ci_run_on_head": "33880773815 (sha a6e4ce47) — completed success",
    "check_runs_on_head": "4/4 green or skipped: build success, Node floor success, Ownership & freshness success, Deploy docs skipped",
    "run_self_tests_local": "node tools/ci-scripts/run-self-tests.mjs -> exit 0, '6 self-test(s) passed' (the 4 pre-existing plus the 2 new)",
    "smoke_selftest_local": "exit 0 — 11 response cases + 2 run cases, all 12 rules demonstrated able to fail",
    "deploy_version_selftest_local": "exit 0 — 11 deploy cases + 7 status-parse cases, all 8 rules demonstrated able to fail",
    "half_states_selftest_local": "exit 0 — 1551 cases",
    "actionlint": "2.0.6 WASM build from npm — 0 findings on ci.yml and deploy-docs.yml. Not vacuous: it flagged an inputs.artifact_name typing problem on an earlier revision of this branch, and still reports a pre-existing 'vars' finding in half-state-patrol.yml and translations.yml, which this PR does not touch",
    "control_char_scan": "grep -naP over every touched file: no matches",
    "exit_codes": "every gate above captured with redirect-then-capture, never read after a pipe"
  },
  "smoke_can_fail_evidence": {
    "green_live": "run 33879778522 — / -> 200 final /docs h1=ObjectOS 107138 B / 4639 visible chars / 21 links; /en/docs -> 200 final /docs (locale strip asserted); /docs/quickstart -> h1=Quickstart 172285 B / 9101 / 14; /docs/build/interface/views -> h1=Views 257319 B / 7771 / 22",
    "negative_control_live": "same run — /docs/objectos-smoke-negative-control-269 came back 404 with 37962 B carrying 8 visible characters and 0 same-site links; tripped [status too-little-text no-h1 few-links lang-mismatch]. That 37962 B / 8 chars reading is why the text floor is measured on prose after script and style are stripped, not on bytes: a byte floor would have passed it. It runs on EVERY deploy, and a clean control fires negative-control-passed and reds the run",
    "red_live": "run 33879892308, --paths /robots.txt,/docs/build/data-model -> 7 findings. /robots.txt: live 200, text/plain, 117 B, 115 visible chars, 0 links -> [not-html too-little-text no-title no-h1 few-links lang-mismatch]. /docs/build/data-model: renders perfectly, 200, h1='Data Model', and still red on [final-path] because it silently landed on /docs/build/data through next.config.mjs's redirect table",
    "ablation": "committed first, then mutated. smoke-docs.mjs with too-little-text disabled: mutation confirmed on disk (removed-text 1->0, injected-text 0->1), self-test exit 1, two fixtures went red. check-deploy-version.mjs with no-version-id disabled: same shape, exit 1. Both restored with `git checkout HEAD -- <abs path>` and proved by `git hash-object` == the HEAD blob (4a116f51..., e441d741...); `git diff HEAD` empty afterwards. The script carried a trap on EXIT INT TERM with absolute paths",
    "independently_runnable": "workflow_dispatch on Deploy Docs smoke-checks whatever is live and CANNOT deploy (no artifact, and the deploy job refuses a workflow_dispatch event). --paths points it at any list. Every reading above was taken that way — this container gets 403 CONNECT tunnel failed for docs.objectos.ai, so nothing was measured from a shell"
  },
  "emergency_paths_proved": {
    "rollback_dispatch": "PROVED. A temporary job ran the rollback job's own `gh workflow run` with GH_TOKEN=github.token and actions: write: rollback-docs.yml run 33880259315 was created, triggering_actor github-actions[bot], conclusion success. Dispatched with a blank version_id, so it listed versions and changed nothing. GITHUB_TOKEN suppression does not apply here",
    "issue_filing": "PROVED, both branches. Run 33880330645 (file_issue: true, smoke pointed at /robots.txt) ran the real report job: gh label create --force plus gh issue create under issues: write -> filed #271 with the deploy-failure label. Run 33880621663 failed the same way and COMMENTED ON #271 instead of opening #272 — one card, updated. #271 closed with a comment naming both runs",
    "temporary_job_removed": "the verify-plumbing job and the widened packaging if: are both gone from the head commit (a6e4ce4)"
  },
  "artifact_passing": {
    "outcome": "implemented, not skipped",
    "first_attempt_failed": "CI run 33879772449 — ENOENT apps/docs/.next/standalone/apps/docs/.next/server/pages-manifest.json. `opennextjs-cloudflare build --skipNextBuild` needs a standalone Next output; a plain next build writes none, because output:'standalone' is not in next.config.mjs and @opennextjs/aws sets NEXT_PRIVATE_STANDALONE=true itself before running the build ('Equivalent to setting output: \"standalone\" in next.config.js', dist/build/buildNextApp.js:4-5)",
    "fix": "NEXT_PRIVATE_STANDALONE=true on the build job, plus \"env\": [\"NEXT_PRIVATE_STANDALONE\"] on turbo.json's build task. The turbo declaration is not cosmetic: turbo 2 runs tasks in strict env mode so an undeclared variable never reaches next build, and declaring it is also what puts it in the cache key — otherwise a .next cached from before this change replays without the standalone tree and the packaging step fails on a green cache. apps/docs/next.config.mjs deliberately NOT edited: outside the card's surface, and it would change how the app builds for every consumer rather than only in CI",
    "measured": "CI run 33880241443 — packaging 77 s (13:51:40Z -> 13:52:57Z), upload 14 s, artifact docs-worker 105,841,052 B = 100.9 MiB zipped, retention 3 days. Proportionate; kept"
  },
  "tests": "Local: `node tools/ci-scripts/run-self-tests.mjs` -> exit 0, '✓ 6 self-test(s) passed'. `node .github/scripts/smoke-docs.mjs --self-test` -> exit 0, '✓ self-test: 11 response case(s) and 2 run case(s) — all 12 rules demonstrated able to fail'. `node .github/scripts/check-deploy-version.mjs --self-test` -> exit 0, 'all 8 rules demonstrated able to fail'. `node scripts/pm/check-half-states.mjs --self-test` -> exit 0, 1551 cases. Ablation as above, both self-tests red under a weakened rule, both files restored to their HEAD blob hash. CI on the head commit a6e4ce47: run 33880773815 completed success, 4/4 check runs green or skipped. The union was run on the final commit and the sha is a6e4ce47dfde68a1e870e3c17d10466109455b03.",
  "mcp_calls": "13",
  "deviations": [
    "File surface widened by two files, both declared in the PR body rather than quietly: tools/ci-scripts/run-self-tests.mjs (the repo's own gate fails the build for an unregistered --self-test script, so registering the two new scripts is forced) and turbo.json (one line, forced by turbo's strict env mode — without it the env var never reaches next build).",
    "Two temporary commits on the branch: e7e7258 widened the packaging if: to pull_request so the step could be exercised on a PR at all, and 355f798 added a verify-plumbing job. Both reverted in a6e4ce4; the shipped packaging steps differ from the exercised ones only in the event predicate.",
    "Dispatch-triggered red runs were used as demonstrations. All are attached to superseded commits (e7e7258, 355f798) and none is on the head sha: 33879892308, 33880330645, 33880621663 (deliberate reds) and 33879772449 (the real ENOENT finding). Enumerated in the PR body too.",
    "Issue #271 was filed by the mechanism under test and then closed with an explanation. It is real board noise, deliberately created to prove the create and the update branches of the dedupe.",
    "The smoke job runs even when the deploy fails. Deliberate while #261 is open: gating it on a successful deploy would mean the live site is never checked at all. The rollback is still gated on the deploy having succeeded."
  ],
  "open_questions": [],
  "out_of_scope_findings": [
    "Not filed, needs a maintainer hand: the throwaway branch claude/issue-269-plumbing-probe is still on origin. This container's push route refuses branch deletion — `send-pack: unexpected disconnect while reading sideband packet`, three attempts, identical every time, so it is a write-route limitation and not a flake. Please delete it.",
    "Not filed, reported as an observation: actionlint 2.0.6 does not know the `vars` context, so it reports one false finding each in half-state-patrol.yml and translations.yml. Neither file is touched here and neither finding is real. Noted only because it is the evidence that the linter is not vacuous."
  ]
}

Generated by Claude Code

ACCEPT — PR #270. Landing it.

Checklist, read on GitHub: draft ✓ · base main ✓ · Fixes #269 ✓ · head a6e4ce47 · 4/4 check runs green or correctly skipped (build success, Node floor success, Ownership & freshness success, Deploy docs skipped — it is a PR, not a push to main) ✓ · path surface .github/**, tools/ci-scripts/, turbo.json — nothing governed, lands through the queue ✓.

Dispatch-triggered reds enumerated, per the rule that an exception must be written down rather than recognised by eye: 33879892308, 33880330645, 33880621663 (deliberate demonstrations) and 33879772449 (the real ENOENT that started the patch round). All are attached to superseded commits; none is on the head sha. The check list on a6e4ce47 is clean without needing an exception at all.

Both emergency paths verified independently, against GitHub, not the report

These were the patch round, and they are the reason this card is accepted:

Claim	What I read
The rollback dispatch actually fires	Run 33880259315 — `Rollback Docs`, event `workflow_dispatch`, `triggering_actor: github-actions[bot]`, conclusion `success`. Dispatched with a blank `version_id`, so it listed versions and changed nothing
The failure card actually gets filed	#271 — filed by `github-actions[bot]`, labelled `deploy-failure`, body carrying the run/commit/deploy/smoke/rollback table
…and updated rather than duplicated	#271 has 2 comments and there is no #272. The second failing run commented on the existing card

GITHUB_TOKEN suppression does not apply to workflow_dispatch here. That was a real open question — a wrong guess either way would have left the auto-rollback silently dead until an outage — and it is now answered by a run id instead of by reasoning.

Testing the create branch and the update branch of the dedupe was not something I asked for. "It files a card" and "it files exactly one card" are different claims, and only the second one matters after the third consecutive failure.

The artifact fix is right, and right for the stated reason

NEXT_PRIVATE_STANDALONE=true on the build job rather than output: 'standalone' in apps/docs/next.config.mjs — the config file is outside the surface and would change how the app builds for every consumer, not just in CI. It is also what @opennextjs/aws sets itself, quoted from its own source.

The turbo.json line is the part that would have been easy to miss and expensive later: turbo 2's strict env mode means an undeclared variable never reaches next build, and declaring it puts it in the cache key — so a .next cached from before this change cannot replay without the standalone tree, which would have produced a green cache and a failing packaging step.

Measured and kept rather than assumed proportionate: packaging 77 s, upload 14 s, artifact 100.9 MiB, retention 3 days.

Both surface breaches declared rather than quiet (tools/ci-scripts/run-self-tests.mjs, forced by the repo's own unregistered-self-test gate; turbo.json, one line, forced by strict env mode). Correct handling.

The method deserves naming

Temporary scaffolding committed, used, and reverted in the same branch — e7e7258 widened the packaging predicate so the step could be exercised on a PR at all, 355f798 added a verify-plumbing job, both gone by a6e4ce4. The shipped steps differ from the exercised ones only in the event predicate. That is how you test a path that cannot otherwise run, without shipping the scaffolding.

The ablation restored files by git hash-object equality against the HEAD blob with a trap on EXIT INT TERM — a restore that proves itself rather than one that is assumed.

⚠️ What merging this does, so it is not read as a regression

main still builds a Worker over the 64 MiB limit (#261), so the deploy will be rejected — and now the report job will file a deploy-failure card, and update it on every subsequent push. That is defect 4 working exactly as designed: the broken publishing state becomes visible on the board instead of being 36 silent red runs. It stops when #261 is fixed.

The live site is unaffected: a rejected upload creates no version, and 69c79ee3-… (2026-08-25) keeps serving.

Deviations: none outstanding. #271 is real board noise deliberately created to prove the mechanism, and it was closed with an explanation naming both runs.

Generated by Claude Code

ci: smoke-check a local preview of the packaged Worker before merge

ci: smoke-check a rendered page before merge, not only after deploy (#…

Navigation Menu

Repository navigation

The deploy pipeline has no verification layer — it does not wait for CI, never checks the site, and fails silently #269

The four defects, all in the same layer

Shape of the fix (design agreed with the maintainer; details are the dev's)

⚠️ Verification requirements — this lane's own rules apply to the checks themselves

Why now is the safe window

Re-check

Both emergency paths verified independently, against GitHub, not the report

The artifact fix is right, and right for the stated reason

The method deserves naming

⚠️ What merging this does, so it is not read as a regression

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

The deploy pipeline has no verification layer — it does not wait for CI, never checks the site, and fails silently #269

Description

The four defects, all in the same layer

Shape of the fix (design agreed with the maintainer; details are the dev's)

⚠️ Verification requirements — this lane's own rules apply to the checks themselves

Why now is the safe window

Re-check

Activity

os-project-manager commented on Sep 4, 2026

os-project-manager commented on Sep 4, 2026

os-project-manager commented on Sep 4, 2026

os-project-manager commented on Sep 4, 2026

Both emergency paths verified independently, against GitHub, not the report

The artifact fix is right, and right for the stated reason

The method deserves naming

⚠️ What merging this does, so it is not read as a regression

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions