Skip to content

feat(release): add opt-in beta provider cohort#537

Open
Gajesh2007 wants to merge 1 commit into
masterfrom
feat/beta-release-cohort
Open

feat(release): add opt-in beta provider cohort#537
Gajesh2007 wants to merge 1 commit into
masterfrom
feat/beta-release-cohort

Conversation

@Gajesh2007

@Gajesh2007 Gajesh2007 commented Jul 12, 2026

Copy link
Copy Markdown
Member

Summary

  • add an opt-in per-provider beta release cohort through darkbloom beta enable / disable, persisted in provider.toml and mirrored into coordinator provider records
  • route manual, startup, background, watchdog, doctor, and banner update discovery through the selected release channel while leaving beta providers in the normal traffic pool
  • introduce stable/beta release persistence, exact SemVer selection, channel-isolated caches and dashboard version reporting, plus strict release/channel validation
  • make beta publication safe: separate R2 pointers, GitHub prereleases that cannot become latest, immutable channel assignment, and verified bundle/binary/metallib hashes
  • keep the old darkbloom beta enable <feature> form for independent local experiments; beta release builds carry their intended defaults instead of requiring fleet operators to toggle every feature

Before

flowchart LR
  subgraph BeforeBehavior[Behavior]
    O[Provider operator] --> F[Enable each beta feature manually]
    F --> R[Restart provider]
    P[Every auto-updating provider] --> L[One global latest release]
    L --> U[Same update target for whole fleet]
    B[Prerelease rollout] --> M[Manual feature flags or global latest replacement]
  end

  subgraph BeforeCode[Code]
    BC[BetaCommand] --> BF[BetaFeatures config fields]
    SU[SelfUpdater call sites] --> EP[GET /v1/releases/latest]
    EP --> GS[GetLatestRelease platform]
    RW[release-swift workflow] --> RP[releases/latest pointer]
  end
Loading

After

flowchart LR
  subgraph AfterBehavior[Behavior]
    O2[Provider operator] --> BE[darkbloom beta enable]
    BE --> CFG[Persist beta channel and enable auto-update]
    CFG --> BP[Beta provider checks beta channel]
    SP[Stable provider] --> SL[Stable-only latest]
    BP --> BL[Highest stable or beta SemVer]
    BL --> V[Existing verify, stage, drain, restart, quarantine and rollback path]
    SL --> V
    V --> T[Provider continues serving normal traffic]
  end

  subgraph AfterCode[Code]
    BC2[BetaCommand] --> PS[ProviderSettings.releaseChannel]
    PS --> SU2[All SelfUpdater constructors]
    PS --> REG[RegisterMessage.release_channel]
    SU2 --> API[handleLatestRelease platform plus channel]
    API --> STORE[ReleaseStore channel-aware selection]
    REG --> PREC[ProviderRecord.release_channel]
    STORE --> HASH[Binary and accepted metallib hash policies]
    WF2[release-swift workflow] --> PTR[Separate stable and beta R2 pointers]
  end
Loading

Safeguards

  • the public/default release channel remains stable; beta discovery must be explicit
  • beta discovery includes stable releases so a final X.Y.Z supersedes X.Y.Z-beta.N
  • -beta versions cannot be registered as stable, stable versions cannot be registered as beta, and a stored release cannot change channels
  • active stable and beta binary/metallib hashes remain simultaneously trusted, so registering a beta cannot deroute stable providers
  • release deactivation refuses binaries still used by connected providers unless explicitly forced
  • leaving beta stops future beta discovery without attempting an unsafe downgrade

Verification

  • go test ./coordinator/...
  • npm test -- src/app/providers (21 tests)
  • npx eslint src/ (0 errors; existing warnings only)
  • swift build --product darkbloom
  • focused Swift beta/config/protocol/updater/watchdog/CLI suite (27 tests)
  • bash -n scripts/admin.sh
  • actionlint .github/workflows/release-swift.yml with the repository's established custom-runner/baseline ShellCheck warnings ignored
  • git diff --check

Rollout

Deploy the coordinator first, then ship this capability in one stable provider release. The first beta must use a newer SemVer core than that bootstrap stable release, for example stable 0.7.9 followed by 0.7.10-beta.1.


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

@vercel

vercel Bot commented Jul 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
d-inference Ready Ready Preview Jul 12, 2026 4:30am
d-inference-console-ui-dev Ready Ready Preview Jul 12, 2026 4:30am
d-inference-landing Ready Ready Preview Jul 12, 2026 4:30am

Request Review

@ethenotethan ethenotethan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Code Review — Layr-Labs/d-inference#

Verdict: REQUEST_CHANGES

Security — ✅ No issues found

Performance — 1 finding(s) (1 blocking)

  • 🟡 [MEDIUM] console-ui/src/app/providers/semver.ts:36 — O(n) array iteration in hot path semver comparison
    • Suggestion: Cache left.length and right.length to avoid repeated property access in the loop

Type_diligence — ✅ No issues found

Additive_complexity — 2 finding(s) (1 blocking)

  • 🟡 [MEDIUM] coordinator/store/memory.go:2830 — validReleaseVersion function duplicates semver.IsValid logic
    • Suggestion: Remove validReleaseVersion and call semver.IsValid directly at call sites
  • 🔵 [INFO] coordinator/api/release_handlers.go:194-196 — Redundant semver validation after regex check
    • Suggestion: Remove the regex pattern check since semver.IsValid provides comprehensive validation

3 finding(s) total, 2 blocking. Verdict: REQUEST_CHANGES.

🤖 Automated review by Centaur · DAR-186

const comparison = comparePrereleaseIdentifier(leftPart, rightPart);
if (comparison !== 0) return comparison < 0;
}
return left.length < right.length;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 [MEDIUM] ⚡ O(n) array iteration in hot path semver comparison

💡 Suggestion: Cache left.length and right.length to avoid repeated property access in the loop

📊 Score: 3×4 = 12 · Category: inefficient_data_structures

if release.Version == "" || release.Platform == "" {
return errors.New("version and platform are required")
}
if !validReleaseVersion(release.Version) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 [MEDIUM] 🧩 validReleaseVersion function duplicates semver.IsValid logic

💡 Suggestion: Remove validReleaseVersion and call semver.IsValid directly at call sites

📊 Score: 3×4 = 12 · Category: over-abstraction

Comment on lines +194 to +196
if !semver.IsValid(canonicalVersion) {
return fmt.Errorf("version must be valid semver")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 [INFO] 🧩 Redundant semver validation after regex check

💡 Suggestion: Remove the regex pattern check since semver.IsValid provides comprehensive validation

📊 Score: 2×3 = 6 · Category: over-abstraction

@github-actions

Copy link
Copy Markdown
Contributor

This PR introduces a beta release channel and mlx.metallib hash verification — neither path weakens any existing mitigation, one open finding (T-025 / SEC-021) is fully resolved, and the new MetallibHashes registry is security-relevant but correctly wired.


Trust boundaries touched

  • TB-006 — release registration, binary hash registry, cache invalidation (release_handlers.go, server.go, store/postgres.go)
  • TB-002 — provider registration message carries new ReleaseChannel field (registry.go)
  • TB-003 — provider config reads release_channel from operator-controlled TOML (ProviderConfig.swift)
  • TB-001handleVersion now passes channel arg to store (consumer.go)

Per-threat assessment

T-025 (SEC-021) — Stale release cache serves revoked binary hashes
Fixes the mitigation.
release_handlers.go line ~146: s.invalidateLatestReleaseCache(release.Platform) replaces the previous s.readCache.Invalidate("latest_release:v1"). This is the exact bug SEC-021 describes — the wrong cache key being invalidated. Confirm invalidateLatestReleaseCache also fires on release deletion (not just registration); the diff only shows the registration path.

T-004 / T-024 (SEC-009) — Release key compared non-constant-time
ℹ️ Neutral. The diff adds channel validation logic and a conflict check (release_handlers.go ~102–108) but does not touch releaseKeyAuthorized / s.releaseKey == comparison. SEC-009 remains open.

T-007 / T-012 / T-027 (SEC-007) — Weight hash enforcement fail-open
ℹ️ Neutral for the existing binary hash path. The new MetallibHash verification in verifyReleaseArtifact (~487–494) is correctly fail-closed: if release.MetallibHash != "" and the metallib is absent or mismatches, registration is rejected. However, if MetallibHash is omitted from the registration request the metallib is silently ignored — this is intentional for backwards compatibility but mirrors the same fail-open pattern as the existing weight_hash omission. Since mlx.metallib is a GPU compute kernel rather than model weights, the security impact is lower than SEC-007, but the same "omit to skip verification" pattern applies.

T-006 — Unauthenticated WebSocket floods provider registry
ℹ️ Neutral. ReleaseChannel is now stored from the provider's self-reported registration message (registry.go ~2532–2536). The comment explicitly says "observational only and never gates routing" — it does not affect routing decisions. No new pre-auth surface is added.

T-015 / T-034 / T-036 — Trust/routing gate integrity
ℹ️ Neutral. The ReleaseChannel field on Provider carries a routing-inert label; the admission logic clamps any unrecognized value to stable (registry.go ~2534–2535), which is the correct fail-safe direction.

T-011 — X25519 key in TOML
ℹ️ Neutral. ProviderConfig.swift gains only a releaseChannel field. No cryptographic material added or removed; T-011 remains open.

T-002 — Unbounded request body
ℹ️ Neutral. The handleVersion change (consumer.go) is a one-line channel arg addition; no body-read path modified.

T-038 — HTTP server config
ℹ️ Neutral. main.go changes are limited to MetallibHashes initialisation and the MergeRuntimeManifest call; http.Server config is untouched.


New attack surface not covered by an existing threat

MergeRuntimeManifest additive merge (server.go ~1414–1456)
The switch from SetRuntimeManifest to MergeRuntimeManifest in main.go means operator-supplied env-var hashes are now added to rather than replacing the release-derived manifest. The comment says "additive emergency control, not a way to silently deroute another active cohort." This is a good design choice but introduces a subtle invariant: if an operator injects a hash via env that was previously blessed and then revoked from the release table, MergeRuntimeManifest will keep it accepted indefinitely until the process restarts without that env var. There is no expiry or revocation mechanism for env-injected hashes. This is consistent with existing behaviour for EIGENINFERENCE_TEMPLATE_HASHES but worth documenting as a deployment note — revocation of a compromised metallib hash requires a coordinator restart with the env var removed, not just a release deactivation.

MetallibHashes set exposed via /v1/runtime/manifest (server.go ~1591)
The accepted metallib hash set is now returned in the public handleRuntimeManifest response. This is consistent with existing exposure of python_hashes / runtime_hashes, but worth noting: the full set of accepted hashes across stable and beta releases is now enumerable, which tells an attacker which beta metallib they need to supply to pass verification.

Beta channel channel-conflict check race (release_handlers.go ~102–108 vs store/postgres.go SetRelease)
The conflict check in handleRegisterRelease calls s.store.ListReleases() and then findRelease() before writing via SetRelease. This is a TOCTOU: two concurrent registrations of the same (version, platform) on different channels could both pass the application-layer check. The DB-level WHERE releases.channel = EXCLUDED.channel on the ON CONFLICT DO UPDATE and the RowsAffected() == 0 guard in SetRelease close this at the storage layer — the application-layer check is defence-in-depth. The DB path is correct; the handler-layer check is redundant but harmless.


Open findings resolved by this PR

  • SEC-021 — per-platform cache key invalidation on release registration is now correct (invalidateLatestReleaseCache(release.Platform)). Verify the same fix is applied on the release deletion path (not visible in this diff).

🔐 Threat model: docs/threat-model.yaml · Updates on each push to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants