Release: reproducible build identity and release manifest (HOM-28, HOM-29)

[Kohandezh]

Summary

HOM-28 (WP01-T04, reproducible backend release artifacts from an exact source revision) and HOM-29 (WP01-T06, machine-readable release manifest), Lane D of the Infrastructure Freeze. An operator can now answer "exactly what source and schema produced this running backend?" (homayar_build_info on /metrics, python -m backend.observability.release show, OCI labels), and release_manifest.py verify turns that into a pass/fail check against the manifest of a commit. A developer's local environment cannot silently change release inputs: the manifest reads only Git objects at the commit, the image build refuses a stale lock, and verify catches an overridden base image or uv version, a different lock/pyproject/migration head and a dirty build context.

Stacked on #43 (HOM-113): #43 is merged (main 87ab41a) and main is merged into this branch (d61cd25, eb9a537); the HOM-28/29 change is 7cc5349 plus the doc commits 1d57995 and 4dd6c73. Lane D merge order is 113 → 114 → 115 → 126 → 28 → 29; this PR merges last. Expected rebase points: ADR 0036 (every Lane D PR appends a decision at the end of the file; a test merge with #47 conflicts only there) and ci.yml (a test merge with #47 auto-merges cleanly; #47 edits the middle of container-config, this PR adds a new job at the end and changes one line of backend).

What changes

• infra/release/release_manifest.py (new, standard library only, Python 3.12)
  • generate --rev <commit>: reads every input with git cat-file / git ls-tree at the commit (ignores GIT_* variables and replace refs; never the working tree). Canonical JSON with sorted keys and no wall clock (build time = commit time unless --build-time). Contents: schema_version; source {repository, sha, tree_sha, commit_time, commit_time_epoch}; application {name, version from Homayar/pyproject.toml}; runtime {python from Homayar/.python-version, base image (reference, digest, pinned_by_digest) and uv version from the Dockerfile ARGs}; dependencies (path + sha256 of Homayar/uv.lock, Homayar/pyproject.toml, Homayar/backend/requirements.txt, Homayar/frontend/package-lock.json, pnpm-lock.yaml, website/pnpm-lock.yaml); schema {migration_head, migration_count} parsed from the revision literals (exactly one head, or it fails; broken chains and duplicate ids also fail); build_inputs per image (Dockerfile and .dockerignore sha256, base images; for api also the release build args and the digest of the shipped code); profile; artifacts.images from --image name=sha256:<digest>; validation from --validation name=url; compatibility.requires_python; reproducibility (what is / is not pinned, computed from the Dockerfiles); manifest_sha256 over the canonical body.
  • build-args --rev <commit>: the KEY=VALUE build arguments of the API image (SOURCE_SHA, SOURCE_DATE_EPOCH, SOURCE_DATE, RELEASE_VERSION, RELEASE_PROFILE, INSTALL_AI_LOCAL).
  • verify --manifest m.json --build-info build-info.json [--image-labels labels.json]: exit 0 OK: ..., or exit 1 with one line per mismatch (manifest_sha256 of an edited manifest, source sha and date, version, profile, uv.lock and pyproject digests, migration head, shipped-code digest and count, base image, uv version, Python minor; with labels: revision, version, created, base.name). Errors exit 2.
• infra/docker/api.Dockerfile: base image pinned by digest (python:3.12-slim-bookworm@sha256:34386ef0…, the multi-platform index resolved 2026-10-18 with docker buildx imagetools inspect, Python 3.12.15); single global defaults for PYTHON_IMAGE, UV_VERSION, INSTALL_AI_LOCAL; uv sync --locked instead of --frozen; build args SOURCE_SHA (default all-zero = unreleased), SOURCE_DATE_EPOCH, SOURCE_DATE, RELEASE_VERSION (default unreleased), RELEASE_PROFILE (default derived from INSTALL_AI_LOCAL; a contradicting value fails the build); OCI labels org.opencontainers.image.{title,source,revision,version,created,base.name}; pyproject.toml and uv.lock copied to /app/release-inputs/; write-build-info writes /app/build-info.json (root-owned, read-only for the app user) before USER.
• Homayar/backend/observability/release.py (from Observability: correlation ids, Prometheus metrics and sweep heartbeats (HOM-113) #43, additive): UNRELEASED_SHA (all zeros) accepted and shown as source_sha="unreleased" in homayar_build_info (label set unchanged); RELEASE_PROFILES (standard, ai-local) validated because the profile is a label; build-info.json gains source.released, build {base_image, uv_version} and code {files_sha256, file_count} (alembic.ini + backend/ without tests, environments and caches — the same rule as api.Dockerfile.dockerignore, implemented identically in the manifest tool and tested for agreement); CLI --base-image, --uv-version.
• CI (.github/workflows/ci.yml): new job release-manifest (needs source-contract, full-history checkout, setup-python 3.12): generates the manifest for ${{ github.sha }} twice and cmps; builds the API image with the build-args output (and checks SOURCE_SHA = github.sha and SOURCE_DATE_EPOCH = git show --format=%ct); copies /app/build-info.json out of the image, dumps the labels, generates the manifest with the image id and the run URL and runs verify --image-labels; starts the image and checks that /metrics reports source_sha="<github.sha>" and prints release show; uploads release-manifest.json, build-info.json, image-labels.json, build-args.txt as artifact release-manifest (actions/upload-artifact@ea165f8d… # v4.6.2, 90 days). Job backend: uv sync --frozen → uv sync --locked. No other job's steps changed.
• Tests Homayar/backend/tests/test_release_manifest.py (19): byte-identical double generation (CLI), manifest describes the revision (tree, version, Python, every lock digest, build args == build-args output), API base image pinned by digest and uv sync --locked only, untracked files / dirty edits / an untracked extra migration head / a planted .env in the working tree do not change the manifest (synthetic repository built from the committed inputs, so the real checkout is never touched), the manifest follows the requested revision, migration head == Alembic ScriptDirectory head == release.code_migration_head() and count == revisions, a broken chain and a second head are refused (CLI exits 2), parser unit tests (annotated literals, merge migrations, duplicates, non-literals), shipped-code rule identical in image and tool, code digest of a clean working tree == the revision's, build_info_document deterministic and validates sha/profile, runtime_identity shows real sha / unreleased / unknown, verify passes on a match and names exactly the mismatching field for source sha, lock digest, migration head, version, code digest and base image, flags an unreleased image and an edited manifest, CLI exit codes and label checks, backend/requirements.txt lists exactly the uv.lock runtime closure (tomllib, extras followed), no secret in the manifest (secret-field names, KEY=value, e-mail addresses, a runtime canary exported as every SECRET_FIELDS env var), validation URLs with credentials/query/fragment and malformed digests/times/revisions refused. Tests skip cleanly without Git metadata, and the working-tree comparisons skip when the shipped code or migrations are uncommitted.
• Docs: docs/operations/release.md (cut a release, generate/verify, read identity from a running backend, what is and is not reproducible, maintenance: re-pin, regenerate requirements, one migration head); index line in docs/README.md; the homayar_build_info row of docs/operations/observability.md (values from build-info.json, unreleased, unknown only outside an image); Release-manifest row and uv sync --locked in docs/operations/ci.md; ADR 0036 Decision 5 appended at the end.

Files outside the list above: none. Not touched: infra/compose/*, infra/scripts/*, backup/restore/tombstone code, migrations, rls.py, saas_models.py, backend/config.py, ReportAppvisionProgress.md.

Local evidence

GitHub Actions is billing-blocked (runs fail at startup with zero steps), so CI has not run; everything below is local. Host: macOS, Docker Desktop linux/x86_64, load average 100–300.

Re-verified on HEAD eb9a537 (main 65f1d75 merged)

• eb9a537: normal merge of origin/main 65f1d75; one conflict in docs/operations/ci.md, resolved by keeping main's console end-to-end row next to this branch's Backend (uv sync --locked) and Release manifest rows. Trailer Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>.
• PostgreSQL 16.15 on 54337: 92 passed, 1 skipped (Redis). actionlint exit 0; gitleaks: no leaks.
• Backend code from main changed the image inputs, so the image was rebuilt from eb9a537 with exactly the build-args output (first attempt): OK: build-info.json matches manifest 85017c43b6b17218 (source eb9a5371077dd0b257e57a45851fa7a0ed504d57, version 0.5.0, migration head d8a2f4c6b1e3); code 238 files; installed_set_sha256 still 7c4769cc…; /metrics shows the commit single-process and multiprocess; the unreleased image still fails verify.
• Note: main's new console end-to-end job installs with uv sync --frozen; it is not part of this PR and was left unchanged.

Re-verified on HEAD 3be6ba8 (after #43 merged; main merged in d61cd25)

• d61cd25: normal merge of origin/main ba70ee4 (includes Observability: correlation ids, Prometheus metrics and sweep heartbeats (HOM-113) #43 as 87ab41a), no conflicts, trailer Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>.
• 3be6ba8: main removed the legacy public site (Remove the legacy public site [3/3] #75: infra/docker/public-web.Dockerfile and the root pnpm-lock.yaml), so generate refused every newer revision; the manifest now covers api, console-web and egress-proxy and website/pnpm-lock.yaml.
• PostgreSQL 16.15 on 54337: test_release_manifest.py + test_observability_metrics.py 92 passed, 1 skipped (Redis).
• Image inputs changed (backend code from main), so the release image was rebuilt from 3be6ba8 with exactly the build-args output (attempt 1 failed on a Debian archive fetch, attempt 2 built). verify --image-labels: OK: build-info.json matches manifest 1ed3ebde9e15aee0 (source 3be6ba89cf12539c4e5998be844e32cee983c03b, version 0.5.0, migration head d8a2f4c6b1e3); code 231 files; installed_set_sha256 still 7c4769cc…; /metrics reports source_sha="3be6ba8…" single-process and in multiprocess mode (/run/homayar-metrics); the unreleased image still fails verify.
• actionlint exit 0, ruff clean, gitleaks: no leaks.

Doc must-fix on HEAD f0b5550 (Integrator)

• manifest_sha256 is described as an integrity checksum, not proof of authorship (unkeyed: whoever edits a manifest can recompute it) in release.md, ADR 0036 decision 5 and the tool docstring.
• Every manifest's reproducibility.not_pinned and release.md now include "not signed / no provenance attestation; images are local ids, not registry digests".
• release.md lists known gaps: verify does not compare the installed distribution set; uv is installed by version, not by hash; the on-prem Compose PYTHON_IMAGE default is a tag; SOURCE_DATE_EPOCH does not control layer file timestamps.
• Evidence: PostgreSQL 16.15 on 54337, test_release_manifest.py + test_observability_metrics.py: 92 passed, 1 skipped (Redis); ruff, ruff format --check and mypy --strict on the tool clean; gitleaks on the changed files: no leaks. Image inputs are unchanged by this commit (docs, a docstring, a message string and one not_pinned entry), so the 4dd6c73 image verification below still describes the build; not rebuilt.

Re-verified on HEAD 4dd6c73 (after merging #43's head e0b577e, Integrator round 2)

Check	Result
Merge origin/backend/hom-113-observability (e0b577e)	merge commit 712e22d, no conflicts. Test merge with origin/main 6694306: clean
Wording follow-up (4dd6c73)	release.py module docstring and ADR 0036 decision 1 consequences now describe build-info.json as written by the API image build (unreleased without release args; unknown only outside an image); the observability.md row already said so (1d57995)
Focused tests on PostgreSQL 16.15 127.0.0.1:54337	test_release_manifest.py + test_observability_metrics.py: 92 passed, 1 skipped (Redis heartbeat, no TEST_REDIS_URL)
actionlint / ruff / gitleaks	exit 0 / All checks passed / no leaks found
Release image	built with exactly the build-args of 4dd6c73, first attempt; Created = commit time 2026-10-18T17:12:26Z; code 211 files ec73f6fc… (= manifest); installed_set_sha256 7c4769cc… (same as all earlier builds)
verify --image-labels	OK: build-info.json matches manifest d1413ddda3dd160b (source 4dd6c730c311ff4ceabd5b27a7ad43c08d879dde, version 0.5.0, migration head d8a2f4c6b1e3); unreleased image: exit 1 with per-field diff
/metrics single process and multiprocess (as compose.cloud.yaml now: read-only, tmpfs /tmp and /run/homayar-metrics, PROMETHEUS_MULTIPROC_DIR=/run/homayar-metrics, 2 workers)	homayar_build_info{…,source_sha="4dd6c730c311ff4ceabd5b27a7ad43c08d879dde",…} 1.0 in both; counters written by uid 10001 under /run/homayar-metrics. The image itself assumes neither path

Re-verified on HEAD 1d57995 (after merging #43's head df191e6)

Check	Result
Merge origin/backend/hom-113-observability (df191e6)	merge commit a5167df, no conflicts (#43 did not touch release.py; ADR 0036 merged automatically, Decision 5 stays last). Test merge with origin/main ed9cd2c: clean
Doc follow-up asked for by #43	docs/operations/observability.md homayar_build_info row: values come from the image's build-info.json; source_sha="unreleased" without release args; unknown only outside an image (commit 1d57995)
Focused tests on PostgreSQL	own cluster: PostgreSQL 16.15 on 127.0.0.1:54337 (fresh initdb, LC_ALL=C, socket /tmp/claude/pgsock-release), TEST_DATABASE_URL=…:54337/homayar_test HOMAYAR_REQUIRE_POSTGRES_TESTS=1 pytest backend/tests/test_release_manifest.py backend/tests/test_observability_metrics.py -v -rs → 88 passed, 1 skipped in 338.8 s (19 release + 69 metrics; skip: Redis heartbeat test, no TEST_REDIS_URL)
actionlint 1.7.7 / ruff / gitleaks v8.28.0	exit 0, no output / All checks passed / no leaks found on the 10 files this PR changes relative to #43
Release image	build-args --rev 1d57995… (SOURCE_SHA and SOURCE_DATE_EPOCH checked against git) → docker build with exactly those args, first attempt (~12 min). Created = 2026-10-18T12:27:16Z (commit time). build-info.json: sha 1d57995…, released: true, code 211 files 0f1ea1fe… (= manifest), installed_set_sha256 7c4769cc… — the same as both earlier builds
verify --image-labels	OK: build-info.json matches manifest 5e02d5283c06b884 (source 1d57995f8f463466b2e659afa521dc21a36207c4, version 0.5.0, migration head d8a2f4c6b1e3); the unreleased image fails with one line per mismatch
/metrics, single process	homayar_build_info{migration_head="d8a2f4c6b1e3",profile="standard",python_version="3.12.15",source_sha="1d57995f8f463466b2e659afa521dc21a36207c4",uv_lock_sha256="78e6e975d3ad0256",version="0.5.0"} 1.0
/metrics, multiprocess like compose.cloud.yaml	--read-only --tmpfs /tmp, PROMETHEUS_MULTIPROC_DIR=/tmp/homayar-metrics, uvicorn --workers 2: same homayar_build_info line; /tmp/homayar-metrics created and written by uid 10001 (counter_*.db, histogram_*.db), so /tmp is writable in the release image as Compose mounts it

Earlier run on 7cc5349 (before merging df191e6)

Check	Command	Result
Focused tests	uv run pytest backend/tests/test_release_manifest.py backend/tests/test_observability_metrics.py -v -rs (one worker, SQLite; the release tests need no database)	58 passed, 1 skipped in 244.9 s (skip: test_redis_heartbeat_store_round_trip, no TEST_REDIS_URL). test_release_manifest.py: 19/19 with no skips, so the working-tree comparisons (code digest, migration head vs Alembic) ran against this commit. #43's metrics tests still pass with the release.py changes. No full-suite run (host load ~300).
Manifest determinism	release_manifest.py generate --rev 7cc5349… twice, cmp	byte-identical, 6124 bytes, manifest_sha256 3734a777…; head d8a2f4c6b1e3, 33 revisions; base image pinned by digest; shipped code 211 files, digest equal to release.code_file_digests() of the clean working tree
actionlint	docker run --rm --volume "$PWD:/repo:ro" --workdir /repo rhysd/actionlint:1.7.7 -color=false .github/workflows/ci.yml	exit 0, no output
hadolint	hadolint/hadolint:v2.12.0-alpine infra/docker/api.Dockerfile (as in CI)	exit 0, no output
ruff	ruff check --config Homayar/pyproject.toml on release.py, test_release_manifest.py, infra/release/release_manifest.py; ruff format --check on the two new files	All checks passed; already formatted
mypy	mypy --config-file Homayar/pyproject.toml backend/observability/release.py backend/tests/test_release_manifest.py ../infra/release/release_manifest.py; mypy --strict infra/release/release_manifest.py	no finding in the three files (the run reports 24 pre-existing errors in untouched modules it follows: backend/config.py, backend/providers/asanak.py, registry.py, whisper.py); strict: Success: no issues found
Lock is current	UV_OFFLINE=1 uv lock --check (uv 0.11.24)	exit 0, Resolved 106 packages — what uv sync --locked enforces
gitleaks	v8.28.0 dir --config .gitleaks.toml --redact on the 9 changed files at HEAD + .gitleaks.toml	no leaks found
Test merges	git merge-tree --write-tree origin/main HEAD; same with origin/backend/hom-126-alerts (#47)	main 4e1b1b0: clean. #47: ci.yml and docs/README.md auto-merge; only ADR 0036 conflicts (both append a decision at the end)
Image without release args	docker build --file infra/docker/api.Dockerfile . (no build args, as the CI smoke test and developers do; built from the working tree during development, before the shipped-code digest was added)	built; uv sync --locked passed in the image; python -m backend.observability.release show: source.sha all zeros, released: false, profile standard, Python 3.12.15, base image python:3.12-slim-bookworm@sha256:34386ef0…, uv 0.11.24, head d8a2f4c6b1e3, 74 installed distributions; labels revision=000…0, version=unreleased, created=1970-01-01T00:00:00Z, base.name = the pinned digest
Image with release args (build-args of this commit)	build-args --rev 7cc5349… (checked SOURCE_SHA = HEAD and SOURCE_DATE_EPOCH = git show --format=%ct), then docker build --file infra/docker/api.Dockerfile with exactly those six --build-args, from the clean checkout	built on attempt 4 (attempts 1–3 failed in uv sync --locked with operation timed out downloading wheels from PyPI under host load, after the lock resolved; SOURCE_DATE_EPOCH disables layer reuse, so each attempt downloads everything; ~29 min). Image config Created = 2026-10-18T07:19:33Z (the commit time). build-info.json: sha 7cc5349…, released: true, date epoch 1792307973, profile standard, Python 3.12.15, base image = the pinned digest, uv 0.11.24, head d8a2f4c6b1e3, code 211 files ce676fff… (= manifest), 74 distributions with installed_set_sha256 7c4769cc… — identical to the unreleased image built hours earlier. Labels: revision 7cc5349…, version 0.5.0, created 2026-10-18T07:19:33Z, base.name = the pinned digest.
CI-job replay against that image	generate --rev 7cc5349… --image api=<image id>, then verify --manifest --build-info --image-labels	OK: build-info.json matches manifest b09ebeb4e96816d1 (source 7cc534915b87649c7bd7ac3017b883e056ba286a, version 0.5.0, migration head d8a2f4c6b1e3)
Negative replay	verify of the unreleased image's build-info.json against the same manifest	exit 1 with one line each for source.sha, source.commit_time_epoch, build_inputs.api.code.files_sha256, build_inputs.api.code.file_count and "built without release arguments (unreleased)"
Running image	docker run as in the CI job (read-only, SQLite, development), curl /healthz, curl /metrics, docker exec … python -m backend.observability.release show	homayar_build_info{migration_head="d8a2f4c6b1e3",profile="standard",python_version="3.12.15",source_sha="7cc534915b87649c7bd7ac3017b883e056ba286a",uv_lock_sha256="78e6e975d3ad0256",version="0.5.0"} 1.0; show prints the same document as the extracted build-info.json

Security notes

• The manifest records digests, versions, paths, the migration head and the commit time only. It never reads environment variables or .env files and records no Git author/committer identity (only the commit epoch). A test exports a runtime-generated canary as every SECRET_FIELDS variable and checks it is absent, and scans for secret-field names, KEY=value pairs and e-mail addresses.
• --validation accepts plain http(s) links only: credentials, query strings and fragments are refused (signed URLs and tokens). --image accepts sha256:<64 hex> only. --rev refuses option-like values and is passed after --end-of-options.
• Git is called with GIT_* variables removed and GIT_NO_REPLACE_OBJECTS=1, so an environment cannot point the tool at another repository, index or object store; blobs are read with git cat-file --batch (no textconv/filters).
• build-info.json is written as root before USER 10001, so the app user cannot rewrite its own identity. No build argument carries a secret.

Honest limits

• Input-reproducible, not bit-for-bit. Same commit + same uv.lock + same base digest + same uv version → same build-info.json and installed distribution set, not the same image digest. Not pinned: Debian packages from apt-get (build-essential, libpq-dev, ca-certificates, ffmpeg, libgomp1, libmagic1, libpq5), file timestamps and layer digests; uv is installed with pip install uv==0.11.24 (version, not hash). build-info.json hashes the shipped code and the names/versions of installed distributions, not their files. The manifest lists all of this under reproducibility.
• Base images of console-web, public-web and egress-proxy remain tags (recorded as pinned_by_digest: false); only the API image was in scope.
• infra/compose/compose.onprem.yaml passes PYTHON_IMAGE: "${PYTHON_IMAGE:-python:3.12-slim-bookworm}" as a build arg, so docker compose build bypasses the digest pin. Compose builds are unreleased anyway (no SOURCE_SHA) and verify reports the base-image difference; changing the Compose default is an infra/compose change I did not make (follow-up for its owner).
• BuildKit exposes SOURCE_DATE_EPOCH to every step, so a release build does not reuse cached layers of an earlier commit (fresh runners in CI are unaffected; local release builds take a full rebuild).
• The CI manifest records the local image id because CI does not push images; a real release records the registry digest (docs/operations/release.md).
• The ai-local profile image (INSTALL_AI_LOCAL=true) was not built locally.
• verify compares the Python minor version with .python-version (the patch comes from the pinned base image).
• Not done here: Homayar/docs/architecture/legacy-retirement.md defers pruning of reportlab (runtime, no importer left) and the ldap3 extra to WP01-T04/HOM-28 (the persian-export extra also has no importer). It changes pyproject.toml, uv.lock and requirements.txt, which the other Lane D PRs share, so it is left for a lead decision (this PR or a follow-up).

[Kohandezh]

Integrator merge evidence (HOM-28, HOM-29), head 693da84:

• Independent reviews: first-round MERGEABLE, with a doc must-fix: the manifest is unkeyed, so not signed and not provenance. That fix is in at f0b5550.
• Delta review of 3be6ba8 (public-web removal) and the main merges: MERGEABLE. Nothing refers to public-web or a root pnpm lock; generate gives byte-identical output; the CI job matches the scripts. test_release_manifest: 19 passed.
• The ADR 0036 conflict with Alerts: actionable Prometheus rules with promtool tests and runbook (HOM-126) #47 is resolved, keeping both sides. Decision 5 wording now says "verify checks that it matches".
• CI on 693da84: 9 of 9 checks pass, including "Release manifest and build identity".
• Known gaps are documented in release.md: verify skips installed_set_sha256, uv is installed by version rather than hash, the PYTHON_IMAGE compose default is a tag, and SOURCE_DATE_EPOCH does not set layer timestamps.
  Merging.