fix(upgrade-check): don't filter upgrade candidates by controller capability#11024
Merged
Conversation
…ability CheckUpgradesAgainst resolved gallery entries through AvailableBackends, which drops every entry the *local* host cannot run. In distributed mode the host running the check is a CPU-only controller while the GPU backends live on worker nodes, so FindGalleryElement returned nil for every cuda/rocm/l4t entry and those backends were silently skipped. Measured on a live cluster: GET /backends reported 48 installed backends, POST /backends/upgrades/check evaluated 5 — all of them plain or cpu-prefixed. The 43 skipped were all hardware-specific builds. As a result cuda13-nvidia-l4t-arm64-longcat-video-development stayed at sha256:0b8dc851 while the registry tag held sha256:38dae6ff, and a cuDNN packaging fix sat unnoticed on a GPU worker for two days. Every name looked up here is already installed somewhere in the cluster, so hardware compatibility was decided at install time; re-deciding it against the controller is wrong. Switch both CheckUpgradesAgainst and UpgradeBackend to AvailableBackendsUnfiltered. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
mudler
added a commit
that referenced
this pull request
Jul 22, 2026
…l replaced (#11029) A backend reinstall could poison every subsequent model load on a worker node until the worker process was restarted. gallery.InstallBackend (and gallery.UpgradeBackend) replace a backend by renaming the live directory to `<name>.install-backup`, moving the staged directory into place, then deleting the backup. A working directory follows the inode across a rename, so a backend process that outlives that swap ends up with a deleted inode as its CWD, and every getcwd(2) in it fails with ENOENT. Observed on a Jetson Thor worker in distributed mode after two successive reinstalls of cuda13-nvidia-l4t-arm64-longcat-video-development. A later model load failed with: rpc error: code = Internal desc = failed to load LongCat model: [Errno 2] No such file or directory The backend's own traceback shows it dying while importing torch, before touching any model file: backend.py line 142 in LoadModel backend.py line 300 in _import_torch torch/_library/custom_ops.py lib._register_fake(...) torch/library.py:183 caller_module = inspect.getmodule(frame) inspect.py:1013 f = getabsfile(module) inspect.py:983 return os.path.normcase(os.path.abspath(_filename)) <frozen posixpath>, line 415, in abspath FileNotFoundError: [Errno 2] No such file or directory os.path.abspath calls os.getcwd() for a relative path. Scanning /proc inside the worker container found the deleted CWD directly: pid 23467 CWD DELETED: /backends/cuda13-nvidia-l4t-arm64-longcat-video-development.install-backup (deleted) Restarting the worker container cleared it (dead CWD count 1 -> 0). Python backends import torch lazily inside LoadModel, so such a survivor still answers HealthCheck and keeps its gRPC port. It looks healthy and only detonates when a model is actually loaded through it. The install paths already stop running processes before replacing the directory (installBackend's force branch, upgradeBackend, backend.delete), but they resolve them by name. That bookkeeping reaps nothing whenever the recorded name no longer resolves into the install's identity set: a legacy entry with an empty backendName, backendIdentity degraded to name-only matching after a ListSystemBackends failure, or an earlier reinstall having already rewritten the metadata.json that carries the alias. Any of those leaves a live process whose directory is about to be unlinked, and nothing downstream notices, because the reuse gate checks liveness and name -- and the name is precisely what does not change across a reinstall. Record the directory each supervised process runs out of, plus that directory's identity at spawn time, and compare with os.SameFile before reusing the process. This needs none of the name bookkeeping to have been correct. Both reuse gates are covered: processMatchesBackend (the install fast path) and startBackend's own already-running branch, which now force-stops such a survivor so the fresh spawn chdirs into the newly installed directory. Processes with no recorded directory are accepted, so a rollout does not restart every running backend once. This matters more with #11024 pending: making GPU backends visible to the upgrade checker will have AutoUpgradeBackends fan upgrades out to worker nodes at scale, and every one of those is a reinstall. Left as is, a rare manual-upgrade footgun becomes a fleet-wide one. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a live distributed cluster,
GET /backendsreported 48 installed backends whilePOST /backends/upgrades/checkevaluated only 5:All 5 are plain/CPU backends. All 43 skipped ones are hardware-specific builds (
cuda13-nvidia-l4t-arm64-*, etc). No GPU backend on the fleet was ever checked for upgrades.Concrete harm:
cuda13-nvidia-l4t-arm64-longcat-video-developmentwas installed at digestsha256:0b8dc851...while the registry tag heldsha256:38dae6ff.... The upgrade check never reported it, and a cuDNN packaging fix sat unnoticed on a GPU worker node for two days until the registry API was queried by hand.Root cause
Not the node inventory.
DistributedBackendManager.CheckUpgradesalready aggregates the cluster-wide installed set (with per-node versions/digests) and feeds it togallery.CheckUpgradesAgainst. The installed side was correct.The gallery side was filtered.
CheckUpgradesAgainstresolved candidates throughAvailableBackends, which appliesGalleryBackend.IsCompatibleWith(systemState)— the controller's system state. On a CPU-only controller,IsBackendCompatiblerejects every name containingl4t/cuda/nvidia/rocm/sycl, so those entries never appear in the candidate list,FindGalleryElementreturnsnil, and the loop hitscontinue. Only names that fall through to "CPU backends are always compatible" survive — exactly the 5 observed.The exclusion is accidental, not deliberate. There is no guard or comment justifying it; the existing tests even work around it (
// Names are kept generic ("my-backend") so the capability filter in AvailableBackends doesn't drop them on a CPU-only test host).AvailableBackendsForCapabilitieswas added earlier to fix the same class of bug for the backend listing endpoint. The upgrade checker never got the equivalent treatment.Fix
Use
AvailableBackendsUnfilteredinCheckUpgradesAgainstand inUpgradeBackend. Every name looked up in either place is already installed somewhere, so hardware compatibility was decided at install time; re-deciding it against whichever host happens to run the check is wrong. Unfiltered is preferred over the capability-union variant because it needs no cluster capability plumbing and cannot under-report.Test
New Ginkgo spec in
core/gallery/upgrade_test.gopins a deterministic CPU-only controller viasystem.NewCapabilityState("default", ...)and asserts an l4t backend installed on a worker at 1.0.0 against a gallery at 2.0.0 is reported. Red before the change:Green after.
go build ./core/... ./pkg/...exit 0,go test ./core/gallery/ ./core/services/nodes/... ./core/application/...exit 0,make lintexit 0.🤖 Generated with Claude Code
https://claude.ai/code/session_0156CbKpDh8qbAYzJZJDZ2yk