Expose the certified dataset SHA-256 and assert the served identity at deploy

[MaxGhenis]

Fixes #742

Part of the UK Autumn Budget CI plan (workstream W4: qualification, promotion and receipts; W4 owns the sim-api deploy and its rollback). Production readiness means three things:

1. the deployed service serves the expected policyengine.py identity;
2. a numerical HTTP sentinel matches;
3. a tested rollback exists.

This PR does (1). #744, stacked on this one, adds (2) and (3).

Changes

• Executor bundle (release_bundle.py). CountryReleaseBundle.default_dataset_sha256 holds the certified SHA-256 of the default dataset, read from the installed bundle in this order:

  1. the certified data artifact's hash, only when that artifact is the default dataset;
  2. otherwise the default dataset's own pin;
  3. otherwise the manifest model.

  Another dataset's hash is never borrowed, and malformed values are dropped.

• Routing state (update_version_registry.py) records default_dataset_sha256 per country. The key is optional, so older readers ignore it.

• Public schema (gateway_models.py):

  • PolicyEngineBundle.dataset_sha256 is optional and must be 64 lowercase hex characters.
  • VersionsResponse.bundles holds the identity of each deployed wrapper version, keyed by wrapper version and then country. It defaults to {}.

• Gateway (endpoints.py):

  • Job responses carry dataset_sha256 for the app's default dataset, which is what dataset names.
  • /versions reports bundles.
  • Raw 202/500 job bodies omit dataset_sha256 when it is unset, so legacy bodies stay byte-identical. This is the rule _bundle_payload already applies to spm.
  • The golden OpenAPI is regenerated.

• Deploy smoke (cloud-run-simulation-entry-smoke.sh). In /versions, each route map holds a version under latest and an app name under each version key. Optional expectations make it prove that:

  • EXPECTED_POLICYENGINE_VERSION: that version has a route and a recorded UK bundle identity;
  • EXPECTED_UK_MODEL_VERSION: that bundle records this UK model, and the model routes to the same app;
  • EXPECTED_UK_DATASET_SHA256: the bundle's dataset digest matches;
  • EXPECTED_LATEST=true: latest routes to that version and model.

  With these unset, behaviour is unchanged. This is the identity the gateway reports from the routing registry; Gate releases on a numerical sentinel and make them reversible #744's numerical sentinel checks what the worker computes.

• Deploy wiring (simulation-deploy.reusable.yml):

  • prepare exports uk_dataset_sha256 from the executor's locked bundle.
  • The candidate smoke and both post-promotion smokes pass the expected identity, but only when the existing stack was deployed, because only then has routing moved.
  • promote_entrypoint now needs prepare.
  • The post-promotion smokes run inside the existing automatic-restore window, so an identity failure after promotion restores the previous revision.

• Updater (check-policyengine-updates.yml, the update script and its helper):

  • The whole dispatch payload is passed in an environment variable and never interpolated into shell.
  • The new verify-dispatch helper command fails the update when the dispatch's bundle.uk (model, core, data package version, dataset SHA-256, certificate digest) differs from the installed wheel's. It fails closed: an installed bundle that cannot report a named field is a mismatch, except for the certificate digest, which older wrappers cannot compute. dispatch_verified is true only when at least one field was compared.
  • A qualification marked promotable: false, or with a verdict that is never promoted, is refused. The verdict and receipt digest are validated before they are printed.
  • Payloads that carry only version keep working.
  • The draft PR body records the dataset SHA-256, the certificate digest, whether the dispatch was verified, and the qualification verdict and receipt digest.
  • The updater still opens a draft PR, as before.

The policyengine.py side of the payload is policyengine.py#574 (the release dispatch). The certificate digest is the receipt-v1 digest from policyengine.py#571; older wrappers report none, and the updater skips it for them.

Invariants

1. No borrowed digest. dataset_sha256 is reported only for the dataset the bundle names as default. It is never a hash of a different dataset. Tests cover the certified, fallback, other-dataset, malformed and manifest-fallback cases.
2. Legacy byte identity. A job body from a worker without the digest has exactly the pre-change key set. This is the existing test_spm_routes set, and a new test covers a worker that has the digest.
3. Identity gate. The smoke fails whenever the release has no route or UK bundle identity, the UK model differs or routes to another app, the dataset digest differs, or (when latest is expected) latest routes elsewhere. It passes with no expectations set. A stub-curl test covers each case. The /versions body in those tests is the gateway's real output: a gateway test regenerates it and fails if the committed fixture drifts (differential).
4. Dispatch binding. The updater opens no PR and makes no commit when the dispatch names a different model, core, data package version or dataset. Matching, legacy and manual payloads proceed. A differential test builds a dispatch from the real locked bundle, the way policyengine.py's release_bundle_identity does, and checks that it verifies.
5. /versions stays up. One historical bundle with an unresolvable dataset is reported without a dataset; it does not fail the response.

Review

Round 1 (an Opus 5.5 agent, with reproductions and mutation runs) asked for changes; all are fixed in bfe10b4.

• High: the smoke compared latest, which is a version, with an app name, so every correct deploy would have failed. My test fixture had the same wrong shape, which is why the tests passed.
• Medium: /versions returned 500 on one bad historical bundle; nothing pinned the workflow wiring; verify-dispatch passed when the installed value was empty.
• Found while fixing: the dispatch's data_release_version was compared with the record's build-style version, so every real dispatch would have failed.

Round 2 (GPT-6.1 Sol) is running.

Tests

• Contract library: 147 passed.
• Gateway: 222 passed.
• Entry: 170 passed.
• Executor: 724 passed, 22 skipped. The skips predate this PR.
• actionlint and shellcheck are clean on the changed workflow and scripts; ruff format --check passes for CI's lint scope.

Not in this PR

• The numerical UK release sentinel.
• Keeping the previous Cloud Run revision after cleanup.
• Reverting Modal routing latest when a run fails after update_routing.
• A manual rollback workflow with a staging drill.
• Expecting latest only when the release should advance it. Until then, an older wrapper deployed without force_latest fails this smoke.

These are in #744.

🤖 Generated with Claude Code