Skip to content

Add build provenance attestation verification - #16

Closed
Gajesh2007 wants to merge 3 commits into
masterfrom
worktree-agent-ae547fbf
Closed

Gajesh2007 wants to merge 3 commits into
masterfrom
worktree-agent-ae547fbf

Conversation

@Gajesh2007

Copy link
Copy Markdown
Member

Summary

  • Adds GitHub Sigstore-based build provenance attestation to the release workflow via actions/attest-build-provenance@v4
  • Coordinator verifies attestations from the GitHub API before accepting release registrations, checking: source repo, workflow path, build trigger, runner environment, and actor
  • Configurable via env vars (GITHUB_ATTESTATION_TOKEN, ATTESTATION_REQUIRED, TRUSTED_ACTORS) with soft mode (warnings) as default
  • Optional gh attestation verify in install.sh for users with the GitHub CLI

Motivation

Currently, a leaked RELEASE_KEY alone is sufficient to register a malicious release with the coordinator. With this change, the coordinator independently verifies that the bundle was built by our GitHub Actions workflow, on GitHub-hosted runners, triggered by an authorized actor — even a compromised release key cannot bypass this.

Changes

File What
.github/workflows/release.yml Added id-token: write + attestations: write permissions, actions/attest-build-provenance@v4 step
coordinator/internal/buildattest/policy.go Policy config loaded from env vars
coordinator/internal/buildattest/github.go GitHub attestation REST API client
coordinator/internal/buildattest/verify.go Sigstore bundle parsing, X.509 Fulcio OID extraction, policy enforcement
coordinator/internal/buildattest/verify_test.go 27 test cases covering valid/invalid bundles, policy enforcement, API mocking, edge cases
coordinator/internal/api/release_handlers.go Attestation verification gate before accepting releases
coordinator/internal/api/server.go Attestation policy field + setter
coordinator/cmd/coordinator/main.go Wiring policy from env vars at startup
scripts/install.sh Optional gh attestation verify (non-blocking)

Deployment

To enable, set on the coordinator:

GITHUB_ATTESTATION_TOKEN=ghp_...  # fine-grained PAT with attestations:read
ATTESTATION_REQUIRED=true          # optional: enforce (default: soft/warn mode)
TRUSTED_ACTORS=Gajesh2007          # optional: restrict release actors

Test plan

  • 27 unit tests covering all verification paths (valid, wrong repo/workflow/trigger/runner/actor, malformed bundles, API errors)
  • Full coordinator test suite passes (go test -race ./...)
  • Builds with CGO_ENABLED=0 GOOS=linux GOARCH=amd64
  • gofmt + go vet clean
  • First real attestation will be created on next tag push — verify coordinator accepts it
  • Test with ATTESTATION_REQUIRED=true in staging before production

Implements GitHub Sigstore-based build provenance attestation to verify
that release bundles were built by the expected CI workflow, repo, actor,
and runner environment. The coordinator verifies attestations via the
GitHub API before accepting release registrations, closing the attack
vector of a leaked RELEASE_KEY being used to register malicious releases.
@vercel

vercel Bot commented Apr 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
d-inference Ready Ready Preview Apr 28, 2026 10:42pm
d-inference-console-ui-dev Ready Ready Preview Apr 28, 2026 10:42pm
d-inference-landing Ready Ready Preview Apr 28, 2026 10:42pm

Request Review

@hankbobtheresearchoor

Copy link
Copy Markdown
Contributor

Hermes Agent Review

Verdict: Minor suggestions

[WARNING]

  • No cryptographic signature verification on DSSE envelopes: VerifyBundle extracts claims from the certificate and parses the DSSE envelope payload, but it does not verify the DSSE signature against the certificate public key. An attacker who can inject a fake attestation bundle could forge claims. The current implementation trusts that the GitHub API returns authentic bundles. Consider adding Sigstore signature verification for defense-in-depth.

  • GitHub API rate limiting without retry/backoff: FetchAttestations returns an error on 429 but the caller does not retry. In production, the coordinator could exhaust the GitHub API rate limit and temporarily block all release registrations. Consider adding exponential backoff with retry for 429 responses.

  • Policy hardcoded to a specific repo: The default TrustedRepo in PolicyFromEnv is hardcoded. While overridable via env var, any fork or alternate deployment must remember to set TRUSTED_REPO. Consider making it a required config parameter instead of relying on a default.

[SUGGESTION]

  • Response body size limit is generous: The 10 MB limit on GitHub API response body is quite large for an attestation list. Consider a tighter limit (e.g., 1 MB) to reduce memory pressure.

  • Log statement in VerifyRelease uses std log: The verified-release log line uses Go standard log.Printf instead of the structured logger used elsewhere. Consider using the structured logger for consistency.

  • CertClaims could validate Issuer: The Issuer field is extracted but never validated against an expected OIDC issuer value. A malicious certificate from a different OIDC provider could pass policy checks if it has matching extensions. Consider adding an issuer allowlist validation step.

[LOOKS GOOD]

  • Comprehensive test coverage: 20+ tests covering valid attestations, wrong repo/workflow/trigger/actor, self-hosted runners, malformed bundles, rate limiting, and edge cases.
  • Policy-based enforcement with Required/Enabled split: The Required vs Enabled distinction is well-designed -- allowing soft enforcement during rollout and hard enforcement once confidence is established.
  • Both V2 (DER) and legacy extension parsing: Backward-compatible claim extraction from Fulcio certificates is thorough.
  • CI workflow attestation step: The attestation generation in the release workflow is correctly placed after the artifact is created, with proper permissions.
  • Install script graceful degradation: Build provenance verification in the install script is optional and non-blocking -- correct for a gradual rollout.

Reviewed by Hermes Agent (hankbob)

cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
Funded attempts must settle (or recovery-review), not refund via release.
Documents DECISIONS #16; race tests cover mark_start vs release and
settle vs release before/after start auth.
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
Aligns chat path with DECISIONS #16 — do not release funded attempts;
emit telemetry and return provider_start_failed_held for recovery review.
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
recover_start_authorized_held returns HeldForReview without moving money,
matching DECISIONS #16 and the HTTP start-failure hold path.
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
Dry-run path for DECISIONS #16: marks a job start_authorized and asserts
HeldForReview without moving money.
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
…ervation

Ops dry-run for DECISIONS #16 force_settle_held: charges a review amount
and restores unused reservation without using the forbidden release path.
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
Eight racing releases on a funded job; all Conflict; hold and balance
unchanged (DECISIONS #16).
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
Eight racing releases on a resize-authorized job; all Conflict; hold
remains (DECISIONS #16).
cursor Bot pushed a commit that referenced this pull request Jul 10, 2026
Release forbidden after start_authorized (DECISIONS #16).
mark_start_authorized on already-authorized job Conflicts (not Ok).
Gajesh2007 added a commit that referenced this pull request Aug 31, 2026
Bump libs/mlx-swift to 6b0505c (fork main, PR #16: vendored MLX/MLX-C at
0.32.2 + regenerated JIT sources) and libs/mlx to 734241bb (fork main,
PR #10: the 0.32.2 core squash carrying the darkbloom patch set).

Content of the new core pin vs the previous darkbloom-base line, verified
against the upstream v0.32.2 tag: identical to v0.32.2 plus the 16-file
darkbloom set (E=256 expert-tile route, Gemma 4 expert-QMM tile kernel,
Metal buffer-count bounds, gpu::eval UAF fix, resource-count telemetry,
thread-safe CompileCacheRegistry). The hand-ported qmv_wide on
darkbloom-base is superseded by upstream's real kernel inside 0.32.2.

What the serving stack gains at B=1-5 (geometry verified from served
checkpoint configs): read-each-K/V-byte-once GQA-8 decode attention
(Qwen 3.5/3.6 35B are 16:2 heads, GPT-OSS 20B is 64:8), a fused
full-attention path for head_dim 256 on NAX (both Qwen 35Bs), qmv_wide/
gemv_wide few-row matmuls, quantized-MoE NAX matmul improvements, and the
raised qmv batch limit on M5-class GPUs, plus the dequantize-in-fp32 and
qmm floor correctness fixes.

libs/mlx .gitmodules branch moves darkbloom-base -> main: PRs #10/#16
made main the integration line; darkbloom-base remains as the legacy
pre-0.32.2 base.
@Gajesh2007 Gajesh2007 closed this Sep 14, 2026

This branch was successfully deployed

4 active (1 outdated) deployments
Preview – d-inference-console-ui-dev — 9c464e2d Deployed Apr 28, 2026 by vercel[bot]
Preview – d-inference — 9c464e2d Deployed Apr 28, 2026 by vercel[bot]
Preview – d-inference-landing — 9c464e2d Deployed Apr 28, 2026 by vercel[bot]
Preview — 52cf682d Deployed Apr 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants