Gate releases on a numerical sentinel and make them reversible

[MaxGhenis]

Draft, stacked on #743 (base w4/release-identity-sentinel). Part of W4 in the UK Autumn Budget 2026 CI plan: production readiness means the expected /versions identity plus a numerical HTTP sentinel, with a tested rollback.

What changes

Numerical release sentinel (src/modal/utils/release_sentinel.py)

• New sentinel_expected job, which runs in parallel with the service deploys. It runs the sentinel request in-process through run_simulation_impl, the worker's own implementation. It uses the exact bundle this deployment installs and the payload the gateway forwards (worker_payload is the gateway's model_dump). It then checks that the result names the deployed policyengine version, UK model and dataset SHA-256.
• The integration job checks that the candidate URL reproduces those numbers; promote_entrypoint checks the stable URL. In both, the request is pinned to the deployed policyengine version. They compare the served bundle identity and budget.{budgetary_impact, tax_revenue_impact, benefit_spending_impact, baseline_net_income}, at relative 1e-9 / absolute £0.01.
• UK requests are not segmented, so the served worker runs the same code on the same payload. A difference means the stack is not serving the bundle that was built: wrong app, wrong data, a stale cache or a broken route.
• The case is basic rate 21% from 1 January 2026, for 2026. Locally on the 6.2.3 bundle it gives £6.68bn in about 11 s with a 2.8 GB peak. I chose 1 January deliberately: on 6.2.3 a 6 April-dated reform gives £0 for 2026 (the dated-reform defect W1 is fixing), and a value that is zero on both sides would prove nothing.
• CI needs PE_UK_PRIVATE_HF_READ_TOKEN and GCP_CREDENTIALS_JSON (the constituency lookup is read from GCS). The environments already hold both.

Routing revert on failure

• update_routing records latest.policyengine before and after publishing, and whether this release should now be latest (newer than the previous latest, or forced). It fails if publishing should have advanced latest but did not.
• The identity smoke (Expose the certified dataset SHA-256 and assert the served identity at deploy #743) requires latest to route to the release only when it should advance it, so an intentional older-version deploy still verifies its own route and identity.
• New revert_routing job: if integration, the authenticated tests or the requested promotion fail (or the run is cancelled), it restores the previous latest and withdraws the failed release. The restore applies only if latest is still that release, so it never overwrites a newer one.
• update_version_registry gains --print-latest and --restore-latest with --expected-latest and --withdraw. Withdrawn country routes go back to the newest remaining bundle with that model version, or are removed.

Rollback target and workflow

• Revision cleanup keeps the revision that served before the deploy (SIMULATION_ENTRYPOINT_ROLLBACK_REVISION), so there is something to roll back to.
• New simulation-rollback.yml (workflow_dispatch). Its inputs are passed through the environment and validated before anything changes. It restores a published release, optionally withdraws one and moves Cloud Run traffic to a retained revision, then runs the identity smoke against the stable URL. It shares the deployments' concurrency group.

Invariants (tested)

• Publishing release R and then restoring the previous latest with R withdrawn returns routes, bundles and latest exactly to their pre-R state. This is a Hypothesis property over release sequences.
• After any restore, every route names an app that still has a policyengine route and manifest, and no route reaches a withdrawn app.
• A conditional restore never changes a state whose latest is something else.
• The sentinel passes only when the served identity matches and every value agrees. Agreement is symmetric and admits only reordering error (Hypothesis). A failed, rejected, missing or late job fails.
• The sentinel's in-process run receives exactly the gateway-forwarded payload (differential test against the gateway model).
• Rollback inputs never appear in shell source; malformed or injection-shaped inputs are refused before any change.

Tests

• executor: 748 passed, 22 skipped.
• entry: 190 passed.
• actionlint and shellcheck are clean on the changed workflows and scripts. The one actionlint report is pre-existing, in pr.yml.
• hypothesis is added to the executor's test extra (lock: one package added).

Review

Independent review (GPT-6.1 Sol) is running at adc244f.

Still to do

• Staging drill after merge. The beta deploy exercises sentinel_expected and the candidate check. Then a manual simulation-rollback.yml run on beta restores the previous release, checks the served identity, and re-promotes. I'll record the run links here.
• Merge order: Expose the certified dataset SHA-256 and assert the served identity at deploy #743 first, then this PR, retargeted to main.
• I'm filing two pre-existing findings separately:
  • budget.households sums household_weight through the weighted-sum helper, giving about 8.3e11 for the UK;
  • on the deployed 6.2.3 pin, April-dated UK reforms score £0 for that year. The policyengine.py side is W1's fix.

🤖 Generated with Claude Code