Skip to content

build(node): generate the eBPF object at build time with a pinned clang instead of committing it (ADR-1622) - #2063

Merged
lusoris merged 6 commits into
masterfrom
build/bpf-object-at-build-time
Oct 5, 2026
Merged

lusoris merged 6 commits into
masterfrom
build/bpf-object-at-build-time

Conversation

@lusoris

@lusoris lusoris commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Summary

The node's eBPF object is now generated at build time with a pinned clang instead of being committed, which removes the ELF that Scorecard's Binary-Artifacts check flags (the maintainer decided this on 2026-10-05). scripts/dev/gen-node-bpf.sh (make node-bpf) is the single generation step; Go CI, docker/Dockerfile.node, dev/Containerfile and the go-build / go-test targets call it. build-config.env pins clang 19.1.7 and the object's sha256; --require-pin refuses anything else, and a missing tool fails naming it, with no fallback and no download.

Stacked on #2061 (it carries the T-SCORECARD-COMMITTED-BPF-OBJECT-2026-10-05 row this PR closes).

The bpf2go binding stays committed source, with its go:embed rewritten to embeddedObject(). The locked Go API Compatibility gate runs go list -export ./... with no generation step, so the package has to compile without the object. A node built without it refuses VMAFX_EBPF_BYPASS=1 and names the generator.

Type

  • build / ci — tooling / infra

Checklist

  • Commits follow Conventional Commits.
  • make format && make lint is green locally (commit hooks passed).
  • Unit tests: not a libvmaf change; go test ./cmd/vmafx-node/... and scripts/dev/tests/test_gen_node_bpf.py pass.
  • No SIMD/GPU code path touched.
  • No feature extractor touched.
  • No new C / C++ source; the new Go and shell files carry the licence header.
  • Not a breaking change for users; contributors need clang to produce the object (migration below).
  • ADR-1622 row lives in docs/adr/_index_fragments/1622-bpf-object-generated-at-build-time.md, slug appended to _order.txt.

Bug-status hygiene

  • docs/state.md: T-SCORECARD-COMMITTED-BPF-OBJECT-2026-10-05 moved from Open to Recently closed with the evidence.

Netflix golden-data gate

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Deep-dive deliverables

  • Research digest — no digest needed: the measurements are in the ADR and the state row.
  • Decision matrix — in ADR-1622 ## Alternatives considered.
  • AGENTS.md invariant note — cmd/vmafx-node/AGENTS.md item 15.
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/changed/node-ebpf-object-generated-at-build-time.md.
  • Rebase note — docs/rebase-notes.md, "The node's eBPF object is generated at build time".

Reproducer

make node-bpf BPF_PIN=--require-pin   # needs clang 19.1.7; without clang it fails naming it
go test ./cmd/vmafx-node/...
python3 -m pytest scripts/dev/tests/test_gen_node_bpf.py
docker build -f docker/Dockerfile.node --target go-builder .

Measured:

  • clang 19.1.7 builds a8079aa4e539dc30e5f275a424f1831d911fb0ef72e579c5c8527b8de2415ca0 on Debian 13 amd64, Debian 13 arm64 (QEMU) and Ubuntu 26.04 amd64; clang 23.1.1 reproduces the previously committed object byte for byte.
  • docker build --target go-builder of Dockerfile.node from a tree without the object: gen-node-bpf --require-pin passes inside the stage and go build produces /out/vmafx-node.
  • With the pinned clang in a golang:1.27-trixie container, TestEmbeddedObjectMatchesPinnedDigest, TestEmbeddedObjectMatchesMirrors and TestEmbeddedObjectLicence pass; without the object the package builds and those tests skip naming make node-bpf.
  • With PATH lacking clang, make node-bpf exits 2: clang not found ... install clang 19.1.7 ... apt-get install clang-19 llvm-19 libbpf-dev.
  • go test ./cmd/vmafx-node/... passes against the in-tree CPU libvmaf (VMAF_BIN set to the CPU vmaf).

Known follow-ups

  • Not verified: Binary-Artifacts reading 10 on the next Scorecard run on master.
  • dev/Containerfile was edited (clang-19 neighbours, llvm-19, libbpf-dev, the generation step) but its full image was not built here; the same script, packages and digest were exercised in the Debian and Ubuntu containers.
  • core/test/fuzz/**/*.bin are the only other .bin files in the tree; they are ASCII text and data, not executables.

Breaking changes / migration

Producing the node's eBPF object needs clang with the BPF target, llvm-strip and the libbpf headers: run make node-bpf (see docs/development/node-ebpf-build.md). Container builds need nothing on the host.

@github-actions github-actions Bot added the type:build Build system / packaging label Oct 5, 2026
lusoris and others added 6 commits October 5, 2026 07:58
…bundles as .sigstore.json (#2061)

* fix(ci): attribute the Metal math headers to Netflix and sign tester bundles as .sigstore.json

Licence Provenance listed seven metal_*_math.h headers (#1921) whose EUPL-1.2
tag disagreed with the Netflix code they reproduce; five now carry the dual tag
and two are recorded as reproducing none. Scorecard Signed-Releases did not
count the tester bundles' .bundle signatures (it counts .sigstore.json), which
with the committed eBPF object took the aggregate to 8.38 against the 8.5
floor; the tester workflows now write .sigstore.json.

* fix(ci): prepend suite venv to PATH in run_affected_suites and isolate hook test python

run_affected_suites.py now prepends the suite venv's bin/ directory to PATH and sets VIRTUAL_ENV so subshells and hook tests execute inside the isolated suite environment, and test_install.py prioritises sys.executable on its PATH.
* fix(ci): start the release workflows for version tags only

Tester prereleases (tester-*, tester-windows-*) fire the release event and
started the production publish, operator-node and supply-chain workflows,
which failed at "Validate tag" and turned master red. Guard each workflow's
first job (and the always() summary jobs) on a v* tag; dependents skip.
Add a contract test over every on: release workflow.
…a licence-gated controller image (ADR-1589) (#2033)

* feat(helm)!: deploy vmafx-controller as its own workload and publish a licence-gated controller image (ADR-1589)

The chart deployed no controller and no workflow built one; a controller
could only run as the server workload with a self-built image and without
its gRPC port on a Service. controller.enabled now renders a one-replica
Recreate Deployment (SQLite queue on a ReadWriteOnce claim) with an http and
a grpc Service port. auth.* is rendered into the controller only, and
auth.enabled and controller.enabled require each other. The nodes and the
operator are pointed at the controller, node.controllerToken and
operator.controllerToken mount their tokens from Secrets, and the
NetworkPolicies open the flows between them.

docker/Dockerfile.controller follows Dockerfile.go-server stage for stage
(the old file linked Debian's libvmaf-dev and had no licence stages), the
licence record production-controller-image reuses the go-server components,
and docker-publish-operator-node.yml builds, signs, SBOMs, smoke-tests and
publishes ghcr.io/vmafx/vmafx-controller with its -source image. The new
GHCR package needs its visibility checked after the first publish. The
controller gains --version (T-CONTROLLER-NO-VERSION-FLAG-2026-10-04).

e2e-k8s.yml bounds its four tool downloads with --max-time and reports
failed diagnostics instead of discarding them; the eleven findings leave the
HISS baseline (413 -> 402; T-E2E-K8S-UNBOUNDED-DOWNLOADS-2026-10-04).
T-HELM-NO-CONTROLLER-WORKLOAD-2026-10-04 closed.

Migration: auth.* now needs controller.enabled. A release that ran a
controller through image.repository fails to render; set controller.enabled
(and controller.image for a custom image) and copy the old queue database to
the <release>-controller-data claim if its jobs are needed.
* chore(deps): Update dependency onnxruntime to >=1.30.0
* chore(deps): refresh the lock fingerprints for the vmaf-tune extras bump
* chore(deps): Update dependency black to v26.10.0
* chore(deps): pin black 26.10.0 in the rc1-tester extras and refresh both black locks

The Makefile pin and the pre-commit hook moved to 26.10.0 in #2044; the
dev-linters lock and the rc1-tester extras were the two places left on
26.5.1. black 26.10.0 leaves the tree unchanged under make format-check.
…ng instead of committing it (ADR-1622) (#2063)

* build(node): generate the eBPF object at build time with a pinned clang instead of committing it (ADR-1622)

OpenSSF Scorecard's Binary-Artifacts check flags the committed
cmd/vmafx-node/bpf/rclonebypass_bpfel.o, which keeps the Scorecard aggregate
under its 8.5 floor. Per the maintainer's decision of 2026-10-05, neither the
object nor its bpf2go binding is committed any more.

scripts/dev/gen-node-bpf.sh generates both and is called from Go CI, the
API-compatibility gate, docker/Dockerfile.node, dev/Containerfile, the Go
Makefile targets and the go vet hook. build-config.env pins clang 19.1.7 and
the object's sha256; --require-pin refuses another clang or digest, and a
missing tool fails naming it. clang 19.1.7 builds the same bytes on Debian 13
(amd64, arm64) and Ubuntu 26.04.

Supersedes the committed-object part of ADR-1539.

Migration: building cmd/vmafx-node now needs clang with the BPF target,
llvm-strip and the libbpf headers; run `make node-bpf` first.
@lusoris
lusoris force-pushed the build/bpf-object-at-build-time branch from 100bf28 to a60b7a9 Compare October 5, 2026 06:16
@lusoris
lusoris merged commit a60b7a9 into master Oct 5, 2026
49 of 57 checks passed
@lusoris
lusoris deleted the build/bpf-object-at-build-time branch October 5, 2026 06:17

This branch was successfully deployed

1 active deployment
github-pages — a60b7a96 Deployed Oct 5, 2026 by lusoris via deploy #4706
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:build Build system / packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant