Skip to content

Add the vendor VFIO vGPU device backend - #364

Open
yummybomb wants to merge 22 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend
Open

Add the vendor VFIO vGPU device backend#364
yummybomb wants to merge 22 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 2 of the vendor VFIO vGPU stack (generalize-vgpu-devicehypervisor-livenessthisvendor-vfio-vgpu). Self-contained in lib/devices + lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).

Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:

  • Discovery & placement — profile discovery from the capacity-dependent creatable_vgpu_types catalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.
  • Release guards — unlike mdev (fresh UUID per assignment), vendor VFIO reuses the same VF path across assignments, so a stale release could clear a later owner's vGPU. Release is guarded by an in-process owner map (covers the window before QEMU opens the device) and an open-VFIO-handle scan (refuses to clear a VF a running VM holds).
  • Reconciliation — clears orphaned assignments on startup, skips VFs in the caller-supplied protected set, and fails closed (skips vendor VFIO entirely) when that set is unavailable, while mdev reconciliation still runs.
  • Per-VF degradation — one unreadable VF is skipped with a warning instead of failing discovery or profile listing wholesale, so a flaky sysfs read cannot blank the host's advertised GPU capacity. Only when no VF is readable does discovery fail, so a wholesale outage cannot demote a vGPU host to passthrough while assignments exist. The open-handle probe stays strict on purpose: it authorizes clearing a reused VF path, so an incomplete scan fails the release rather than risk a false "not in use".
  • Integration test — branched by discovered framework, extended to cover release on stop and reacquisition on start.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/resources/ pass

Note

Medium Risk
Touches host GPU mode detection, sysfs assignment/release, and /resources admission paths; vendor VFIO release logic is security-sensitive (false “not in use” could clear another owner’s VF), though create is gated and release fails closed on incomplete VFIO scans.

Overview
Introduces dual vGPU frameworks (mdev vs vendor VFIO via current_vgpu_type / creatable_vgpu_types) with DiscoverVGPU, framework-aware profile listing, DestroyVGPU/ReconcileVGPUs, and a large Linux vendor VFIO implementation (placement, owner + open-VFIO release guards, orphan reconciliation, create rollback).

Removes DetectHostGPUMode; resources now derive vGPU vs passthrough from DiscoverVGPU (with context) and avoid mis-reporting passthrough when vGPU discovery fails transiently. Mdev discovery skips unreadable VFs instead of zeroing capacity; instance vGPU teardown passes InstanceID on assignments.

CreateVGPU still rejects vendor VFIO until durable lifecycle integration; integration test skips vendor VFIO hosts accordingly but adds framework-aware sysfs checks and stop/start release/reacquire for mdev.

Docs/OpenAPI/metrics clarify ephemeral vGPU lifecycle, QEMU-only practical vGPU support, best-effort profile availability, and vendor VFIO wedge troubleshooting.

Reviewed by Cursor Bugbot for commit 1f1f24f. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 2bb8e86 to 7fc3b49 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7fc3b49 to f661e63 Compare August 6, 2026 19:40
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from f661e63 to d1207d0 Compare August 7, 2026 14:02
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from d1207d0 to 5b47670 Compare August 7, 2026 15:04
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 5b47670 to ffbf8a0 Compare August 7, 2026 20:52
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from ffbf8a0 to 78801e7 Compare August 8, 2026 01:05
Comment thread integration/vgpu_test.go Outdated
@yummybomb

Copy link
Copy Markdown
Contributor Author

added 7f5233f: report GPUProfile.Available as the count of free VFs advertising the profile type (creatable-instance units), matching mdev's summed available_instances, the OpenAPI description, and the integration test's decrement assertion. note the gpuProfileSlots metric steps up on vendor VFIO hosts (per-GPU → per-free-VF units). go test ./lib/devices ./lib/resources green; the hardware integration test was not run locally.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7f5233f to 0baacb3 Compare August 10, 2026 07:26
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

116 files generated at dd54e02 (pushed)

go get github.com/kernel/hypeman-go-staging@dd54e02078ec70a34c48b149e0080ba65dde4c01
python code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

230 files generated at 1e0016e (pushed)

typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

138 files generated at 5a2fd33 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_769WR6Ls-thankful-shard
Timestamp2026-08-18T14:46:37.661Z
stlc8413509
Spec hash101b9ae4ce99
Config hash55e15f6f4434

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-18 14:47:10 UTC

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from bc7c23f to cb635dd Compare August 17, 2026 15:00
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from cb635dd to bb5b6ea Compare August 17, 2026 15:23

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • mdev discovery now hard-fails on one unreadable mdev_supported_types, zeroing host GPU capacity — vendor VFIO degrades per-VF. intentional asymmetry? Also that is different behavior than before the VFIO changes.
  • reconcile silently skips an allocated + in-use + unprotected VF. log at warn level.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from c3b37b1 to dae5383 Compare August 17, 2026 21:34
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are
assigned by writing a type ID to a VF's nvidia/current_vgpu_type and
passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend
behind the existing framework dispatch: profile discovery from the
capacity-dependent creatable catalogs, least-loaded VF placement,
create/verify/rollback, and release.

Because the same VF path is reused across assignments (unlike mdev
UUIDs), release is guarded: an in-process owner map covers the window
before QEMU opens the device, and an open-VFIO-handle scan refuses to
clear a VF a running VM still holds. Reconciliation clears orphaned
assignments on startup, skipping VFs protected by the caller and
failing closed when the protected set is unavailable.

Branch the vGPU integration test by discovered framework and extend it
to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
The instance lifecycle already routes create/start/stop/delete through
CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would
activate the backend before assignment durability and release guards
exist. Reject vendor VFIO creates for now; destroy stays wired so
existing assignments remain releasable. The integration test skips on
vendor VFIO hosts at this layer and no longer asserts the transitional
stop-retention behavior.
Counting every free VF advertising a type overreports concurrent
capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q
assignment revokes the type from every other VF on that GPU. Bound each
GPU's contribution by both its free VFs and how many times the profile
framebuffer fits into the GPU's remaining framebuffer, using the largest
still-creatable profile as a lower bound on what remains.
A single unreadable current_vgpu_type failed discoverVFs wholesale, and
GetGPUStatus turns a discovery error into a host with no GPU, so one
flaky sysfs read blanked out the host's entire GPU capacity for
admission and monitoring.

Skip unreadable VFs with a warning and keep the readable inventory: a
skipped VF is never selected for placement and never reconciled, both
safe directions. When no VF is readable, discovery still fails so a
wholesale sysfs outage cannot demote a vGPU host to passthrough while
assignments exist.

Also document that vendor VFIO vGPUs are known broken on Cloud
Hypervisor upstream and QEMU is the required hypervisor for GPU
instances.
listProfiles failed wholesale when one VF's creatable_vgpu_types read
failed, blanking every advertised profile while discoverVFs directly
above it already skips unreadable VFs for exactly that reason. Skip and
warn instead; underreporting is the safe direction for status and
admission.

Also document why openVFIOPaths stays strict where mdev's scan is lax
(it authorizes clearing a reused VF path), and the 0Q/0B parsing caveat
in framebufferFromProfileName.
listProfiles skips an unreadable VF but create still failed placement
wholesale when profileMetadata or selectLeastLoadedVF hit the same VF,
so /resources could advertise capacity a create then failed to use.
Skip the VF in both loops; it simply stops being a placement candidate.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from dae5383 to 1f1f24f Compare August 18, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants