Fingerprint policyengine.py 6.x's bundle manifest in PolicyEngine provenance

[MaxGhenis]

Summary

Fixes two findings from the pre-merge delta review of #182 (merged as d616e67).

A. The provenance fingerprint missed policyengine.py 6.x's bundle manifest. policyengine_provenance_inputs() hashed only policyengine/data/release_manifests/{country}.json. policyengine.py 6.1.2 ships none of those. It ships one policyengine/data/bundle/manifest.json, which _load_raw_policyengine_manifest falls back to (data_releases.<country>), and which policyengine's own get_release_manifest also reads in 6.1.2. Under the pinned env the fingerprint recorded {'us': None, 'uk': None}, so an in-place edit of the bundle manifest between the provenance writer and a worker went undetected. The fingerprint now also records bundle_manifest_sha256, which is None when the file is absent. Both manifest paths are now module constants that the reader and the fingerprint share, so they can't drift apart again. The docstring no longer claims the manifests are "recorded directly" without qualification. It lists what is recorded, in either layout.

B. The supervisor docstring overstated the cost of computing provenance. It said computing provenance "imports policyengine, which builds the US and UK tax-benefit systems". That happens only when a country's installed model is the exact version policyengine.py pins. The docstring now describes both cases. Computing once per run still bounds the cost when the import does happen.

Verified this session

• 6.1.2 layout: site-packages/policyengine/data holds only bundle/manifest.json and schemas/. Before the fix: release_manifest_sha256 == {'uk': None, 'us': None}.
• 4.16.1 layout (another worktree's venv): per-country data/release_manifests/{us,uk}.json, and no bundle/manifest.json (its data/bundle/ holds bundle.json). Its get_release_manifest reads the per-country files.
• Reviewer's demo on the real 6.1.2 manifest (this worktree's venv; file backed up, then restored to its original sha256 d7e6308e…):
  1. Write provenance.
  2. Edit data_releases.us.bundle_id in place.
  3. A fresh process now reports PolicyEngine provenance file not reused (…): it was written for different PolicyEngine packages or code and resolves us-6.1.2-EDITED.
• No import under the pin: uv run python -c "import sys; from policybench.policyengine_runtime import policyengine_bundles_for_countries as f; f({'us'}); print([m for m in sys.modules if m.startswith('policyengine')])" prints [], and so does {'us','uk'}. policyengine 6.1.2 pins policyengine-us 2.2.1 (2.15.17 installed) and policyengine-uk 2.90.2 (2.89.0 installed). The versions differ, so _load_policyengine_manifest is never called.
• Import under 4.16.1: it pins policyengine-us 1.723.0, which is installed. The same call loads policyengine, policyengine_core, policyengine_us.system and policyengine_uk.system, and took 11.1 CPU-s at 909 MB max RSS. That matches the dated 2026-09-28 figure kept in the docstring.

Tests (tests/test_policyengine_runtime.py)

• test_manifest_edited_in_place_makes_the_provenance_file_stale[per-country|bundle]: the reviewer's in-place bundle_id edit, run on a fake layout built from the existing test_raw_manifest_reads_the_policyengine_6_bundle_layout pattern. _read_policyengine_provenance refuses the stale file, and resolve_policyengine_bundles recomputes the edited release.
• test_provenance_inputs_hash_each_layouts_manifests[per-country|bundle]: the per-country layout records each file's sha256 as before, with bundle_manifest_sha256 = None. The bundle layout records the bundle hash, with None per country.
• test_installed_policyengine_manifest_is_fingerprinted: the pinned policyengine.py's manifest is actually hashed.
• test_equal_fingerprints_mean_equal_raw_manifests (Hypothesis) covers any mix of the two layouts' files, each present or absent.

Invariants

• Fingerprints are equal exactly when the manifest files are byte-identical, wherever they are installed. So equal fingerprints hand every country the same raw release.
• The fingerprint is byte-level. Reordering a manifest's keys changes it even though the parsed release is equal. That is intended: it can only make a worker recompute, never reuse a stale file.

Mutation check: against the old policyengine_runtime.py, the bundle-layout stale test and the property test fail on every seed tried. The per-country tests fail only on the new key; their per-country hashes still match.

Gates run locally

• uv run ruff check .: passes.
• uv run ruff format --check .: passes.
• uv run pytest -q -m "not slow": 1056 passed, 5 skipped, 13 deselected.

Interaction with open PRs

#184 (chunked-provenance-handoff) also edits policyengine_runtime.py and supervisor.py, and both auto-merge with this branch. #184 already conflicts with main in tests/test_supervisor.py, because #182 changed that file after #184 branched. This PR doesn't touch that file.

axiom: n/a: provenance fingerprint and docstring; no policy change.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Sep 30, 2026 12:45pm UTC

[MaxGhenis]

Review record before merge

Independent reviews: all APPROVE, nothing blocking

• Opus 5.5 on a Subfleet lane (20260930-082027-pb185-review-opus), reviewing 6cbbe32f. It read the change without a shell, so it ran nothing itself; its unexecuted checks were run by the author and by the execution verifier below.
• In-session Opus 5.5 execution verifier, reviewing 69353b68. It ran:
  • ruff, and tests/test_policyengine_runtime.py (34 passed), plus the Hypothesis property under 8 seeds and 2×3000 examples;
  • old origin/main runtime against the new tests: 5 fail, including the bundle-layout stale test and the property test;
  • four extra mutants, all caught;
  • the no-import one-liner: [];
  • the real bundle hash compared with shasum: equal;
  • the reviewer's in-place bundle_id demo in a scratch copy. New code refuses the file; old code serves the stale us-6.1.2.
• In-session Opus 5.5 delta reviewer, reviewing 69353b68. It checked every docstring and comment claim against the 6.1.2 and 4.16.1 sources and METADATA, confirmed that nothing else reads inputs, and found that the only conflict with Compute PolicyEngine provenance once per chunked or retry invocation #184 is the one in tests/test_supervisor.py, which Compute PolicyEngine provenance once per chunked or retry invocation #184 already has against main.
• A GPT-6.1 Sol review was cancelled before launch because Codex lanes have no capacity until 2026-10-03.

Follow-ups applied

• 69353b68: qualify the POLICYENGINE_SKIP_COUNTRY_IMPORTS condition, make the runbook say "instead of computing it", and add test comments.
• 7b8c0c21: add explicit Hypothesis @examples for the mixed layout and for whole-file hashing, with dataset_overlays added to the strategy. A mixed-layout mutant had survived some seeds, and a mutant that hashes only data_releases was caught only by the exact-hash test; both are now caught on seeds 1 and 5. It also gives the direct-computation test ids that don't depend on PYTHONHASHSEED.

Known limits, unchanged by this PR and not reachable under the current pins

• The fingerprint locates manifests via metadata.distribution().locate_file, while policyengine's own loader uses importlib.resources.files. These differ only if another policyengine package shadows site-packages.
• In the importing branch, default_dataset_uri can fall back to resolve_dataset_reference, which checks a path relative to the working directory or fetches from Hugging Face. Neither is fingerprinted. That branch runs only when an installed model is the exact version policyengine.py pins, which is not the case under 6.1.2.

CI on 7b8c0c21: app, lint, paper, test and Vercel all pass.

🤖 Generated with Claude Code