Skip to content

feat(meta): VMAFX Phase 4b distributed platform umbrella ADR - #21

Merged
lusoris merged 1 commit into
masterfrom
feat/vmafx-phase4b-distributed-platform-adr-0709
May 28, 2026
Merged

lusoris merged 1 commit into
masterfrom
feat/vmafx-phase4b-distributed-platform-adr-0709

Conversation

@lusoris

@lusoris lusoris commented May 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Files ADR-0709 — the Phase 4b umbrella architectural decision record locking the VMAFX transformation from single-binary scoring tool into a distributed video-quality, encoding, and ML platform.
  • Adds docs/architecture/phase4b-distributed-platform.md with a Mermaid component diagram showing the controller/node/operator layout, CRD summary, storage flow, and GPU pool affinity table.
  • Updates ADR README index, docs/state.md, docs/rebase-notes.md, and changelog.d/.

Locked decisions (ADR-0709)

  • Architecture: vmafx-controller (Go) + vmafx-node (Go) + vmafx-operator (Go, controller-runtime) + thin clients (vmafx-mcp, vmafx-tune, CLI)
  • ffmpeg: latest pinned release, bundled in node distroless layer; existing ffmpeg-patches/ stack applies in-container
  • Storage: rclone-mount / rclone-vfs zero-copy POSIX view of S3 / GCS / Azure Blob / SFTP
  • eBPF: research-first — one concrete optimization target after measuring baseline
  • AI inference: Go ONNX Runtime (onnxruntime-go) inside vmafx-node; CUDA EP + ROCm EP + OpenVINO EP
  • Sidecar training: Python sidecar container (v1) — captures (ref, dis, score, metadata) triples during encoding for continuous model refinement
  • C ABI break: public API rewrites to C++23 + Rust + Go surface; ffmpeg-patches/ updated in the same Phase 4b.8 PR
  • Native build sunset: Docker images + Helm chart are the only user-facing release artifacts; no external .so / .deb / .rpm packages

Implementation plan (Phase 4b.N)

Phase Scope
4b.1 vmafx-server → vmafx-controller (job queue, node registry, scheduler)
4b.2 vmafx-node Go binary (libvmaf cgo, ffmpeg subprocess, Go ONNX Runtime)
4b.3 vmafx-operator kubebuilder skeleton + CRDs (VmafxJob, VmafxNode, VmafxModelTraining)
4b.4 ffmpeg latest bundled in node distroless layer
4b.5 rclone integration (node layer + mount lifecycle)
4b.6 eBPF research digest + ONE concrete optimization
4b.7 Sidecar training v1 (Python sidecar + triple-capture API)
4b.8 C ABI break + ffmpeg-patches update
4b.9 Native build sunset (Docker + Helm only release artifacts)

Test plan

  • All pre-commit hooks pass (conventional commit, ADR collision check, no large files, no secrets)
  • ADR number 0709 claimed via scripts/adr/next-free.sh --claim vmafx-phase4b-distributed-platform
  • No code changes in this PR — doc-only; no build or test gate applies
  • docs/rebase-notes.md entry confirms no upstream rebase impact

Deliverables checklist

  • (1) research digest: no digest needed: umbrella ADR — architectural decisions only, no new algorithm or implementation
  • (2) decision matrix: in ADR-0709 ## Alternatives considered (six separate tables covering architecture, ffmpeg integration, storage, eBPF scope, AI inference, sidecar training, C ABI break, and native build publishing)
  • (3) AGENTS.md invariant note: no rebase-sensitive invariants — doc-only PR
  • (4) reproducer / smoke-test: grep -c "Phase 4b" docs/adr/0709-vmafx-phase4b-distributed-platform.md returns non-zero; ls docs/architecture/phase4b-distributed-platform.md exists
  • (5) changelog.d/added/vmafx-phase4b-umbrella-adr.md fragment
  • (6) docs/rebase-notes.md — no rebase impact: doc-only, no C/Go/Rust source changed

state.md

T-VMAFX-PHASE4B-ADR-0709 row added to docs/state.md.

🤖 Generated with Claude Code

@lusoris
lusoris marked this pull request as ready for review May 28, 2026 11:37
Files:
- docs/adr/0709-vmafx-phase4b-distributed-platform.md  (ADR-0709, Proposed)
- docs/architecture/phase4b-distributed-platform.md    (Mermaid component diagram)
- changelog.d/added/vmafx-phase4b-umbrella-adr.md
- docs/adr/README.md  (index row for ADR-0709)
- docs/state.md       (T-VMAFX-PHASE4B-ADR-0709 row)
- docs/rebase-notes.md (no rebase impact: doc-only)

Locks the Phase 4b architecture: controller/node/operator component split,
ffmpeg worker integration (latest ffmpeg), rclone zero-copy storage
(rclone-mount / rclone-vfs), eBPF research-first path, Go ONNX Runtime AI
inference inside vmafx-node, Python sidecar continuous training (v1),
C ABI break with ffmpeg-patches update (Phase 4b.8), and native build
sunset — Docker images + Helm chart are the only release artifacts going
forward.

Nine-phase implementation plan (4b.1 through 4b.9) establishes dependency
order for per-sweep child ADRs and PRs.

No code changes in this PR. Per-sweep implementation PRs follow in their
own dispatches.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the feat/vmafx-phase4b-distributed-platform-adr-0709 branch from 19ccd13 to 12efcdf Compare May 28, 2026 11:43
@lusoris
lusoris merged commit ca54089 into master May 28, 2026
11 of 21 checks passed
@lusoris
lusoris deleted the feat/vmafx-phase4b-distributed-platform-adr-0709 branch May 28, 2026 11:43
lusoris added a commit that referenced this pull request Jun 28, 2026
… NULL-guard QSV chain (#1067)

Two verified defects from the adversarial round-4 audit, in the dedicated
libvmaf_sycl filter added by ffmpeg-patches/0005.

- sycl_state leak (high): uninit_sycl called vmaf_close() but never
  vmaf_sycl_state_free(). vmaf_close() explicitly does NOT free the SYCL state
  (ownership is not transferred), so the state + its USM allocations leaked on
  every filter close. Now freed explicitly after vmaf_close(); the false
  "vmaf_close handles it" comment is removed.
- QSV NULL deref (med): do_vmaf_sycl walked the QSV zero-copy handle chain
  (AVFrame->data[3] -> mfxFrameSurface1* -> Data.MemId -> mfxHDLPair* ->
  ->first) with no NULL guards; a software-fallback QSV surface with no VA
  backing crashed the filter. All three chain links are now NULL-checked,
  returning AVERROR(EINVAL) with a clear diagnostic.

The patch was regenerated surgically (only these two + blocks in the
vf_libvmaf.c hunk + the hunk-header recount 335->353; the configure/Makefile/
allfilters hunks are byte-unchanged). Verified by a full 16-patch
`git apply --3way` series replay against n8.1.1 (CLAUDE rule #14).

Finding #21 (a redundant but idempotent `check_pkg_config libvmaf_sycl`
configure probe) is intentionally left: the probe is idempotent and removing
the wrong one risks breaking SYCL detection.

Co-authored-by: Lusoris <lusoris@pm.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@lusoris lusoris added this to the 1.0.0 — First release milestone Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant