[pull] master from mudler:master#1202
Merged
Merged
Conversation
…ef6f` (#11010) * ⬆️ Update antirez/ds4 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(ds4): link new tensor parallel objects The updated ds4 revision split tensor-parallel transport and layer placement into separate translation units. Build and link those objects on CPU, CUDA, and Metal builds. Assisted-by: Codex:gpt-5 --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
…ability (#11024) CheckUpgradesAgainst resolved gallery entries through AvailableBackends, which drops every entry the *local* host cannot run. In distributed mode the host running the check is a CPU-only controller while the GPU backends live on worker nodes, so FindGalleryElement returned nil for every cuda/rocm/l4t entry and those backends were silently skipped. Measured on a live cluster: GET /backends reported 48 installed backends, POST /backends/upgrades/check evaluated 5 — all of them plain or cpu-prefixed. The 43 skipped were all hardware-specific builds. As a result cuda13-nvidia-l4t-arm64-longcat-video-development stayed at sha256:0b8dc851 while the registry tag held sha256:38dae6ff, and a cuDNN packaging fix sat unnoticed on a GPU worker for two days. Every name looked up here is already installed somewhere in the cluster, so hardware compatibility was decided at install time; re-deciding it against the controller is wrong. Switch both CheckUpgradesAgainst and UpgradeBackend to AvailableBackendsUnfiltered. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…all (#11026) Testing the progress-based cold-load deadline on the live cluster surfaced a false positive. The stall window observed UPLOAD bytes only, but the staging path has a phase that does real work while moving zero upload bytes: the resumable-upload verify phase. When a shard is already present on the worker from an earlier attempt, the frontend HEADs it, hashes the local copy to confirm it matches, and skips the transfer. Staging a 70 GB model with 56 GB already staged: 17:27:34 INFO Upload skipped (file already exists with matching hash) ... 17:28:20 INFO Upload skipped (file already exists with matching hash) ... 17:29:07 INFO Upload skipped (file already exists with matching hash) ... ... six-plus consecutive minutes, no bytes uploaded at all ~45s per skipped ~4 GB shard. That is correct and desirable - it is what makes resume work - but it was indistinguishable from a stall. At 45s per shard it sits inside the 5m window, so the run in flight was fine; the problem is the 600 GB scale this machinery exists to enable, where one shard can plausibly hash for longer than the window. The guard would then fire during verification of a transfer that is working perfectly. Verified mechanism: probeExisting() HEADs the worker and then calls downloader.CalculateSHA(). The staging progress callback is only consulted inside doUpload(), which the skip path never reaches, so observeLoadProgress was called zero times for the whole verify phase. Verification exposed a second, worse bug in the same path: CalculateSHA consults no context at all. An expired cold load kept hashing to completion, compared the hashes, and returned success - reporting a file as staged on a dead load. The failure only surfaced on the NEXT file, whose HEAD died immediately. That is exactly the shape of the red test here, which fails on shard 3. Fix: hash in 1 MiB chunks via hashFileWithActivity(), ticking the cold-load deadline per chunk and checking ctx per chunk. A successful HEAD also counts, since a 200 with a content hash proves the worker is serving right now. Counting hash progress does not make a dead transfer look alive: hashing is bounded, terminating work proportional to file size, in probeExisting it runs only after a HEAD proved the worker was up, and the 24h absolute cap still bounds the whole hold. The alternative of simply widening the window was rejected - it would reintroduce the size cliff this work removes. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… variant ladders (#11027) * fix(gallery): repoint qwen3-4b/qwen3.5-9b dflash drafters at post-rename GGUFs The drafters both entries referenced were converted from the pre-merge DFlash PR branch and carry dflash.target_layer_ids. llama.cpp reads dflash.target_layers and refuses the load. The stored values are offset by +1 relative to the HF-side field, so the files cannot be repaired by renaming the key and must be replaced. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): add apexentries HuggingFace client Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): build the HF client via pkg/httpclient The apexentries HuggingFace client was constructed as a raw &http.Client{Timeout: 60s}. The repo convention (documented in .golangci.yml, which cannot express this as a forbidigo pattern) is that all outbound HTTP goes through pkg/httpclient, which refuses redirects by default and sets a TLS 1.2 floor. The std client follows redirects and forwards custom credential headers to the redirect target on a cross-host hop (GHSA-3mj3-57v2-4636). Only a User-Agent is sent today, but this calls an external API and an HF_TOKEN header added later would leak. Switch to httpclient.NewWithTimeout, preserving the 60 second timeout. No behaviour change for the current header set. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): discover APEX tiers by filename suffix Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): resolve unsloth counterparts and sharded quants Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): render APEX child entries with the dflash/mtp tag rule Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): set backend, known_usecases and cross-repo drafters RenderChild left three gaps against the hand-written gallery entries. The generated entries reference gallery/virtual.yaml, which supplies no backend, so every generated entry named no engine at all. All comparable hand-written entries set backend: llama-cpp in overrides; do the same. Set known_usecases to [chat] alongside it: LocalAI falls back to the backend defaults when it is absent, so this is convention rather than breakage, but generated entries should not read differently from their neighbours. The drafter was also assumed to live in the repo publishing the weights. Speculative pairings routinely cross repos, and a drafter URI built from the weights repo 404s at install time. Add ChildInput.DraftRepo, used for both the drafter URI and its local path, falling back to Repo when empty so pairings that do ship the drafter alongside the weights are unchanged. The dflash/mtp tagging rule is untouched: the tag still follows SpecType and nothing else. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): dedupe generated entries against the existing gallery Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * apexentries: canonicalize HF URIs and dedup the generated batch Merge exists to stop a second gallery entry being added for weights the gallery already ships, but two gaps let duplicates through on a bulk run. The URI key was compared as an exact string while render.go only ever emits https://huggingface.co/{repo}/resolve/main/{file} and the gallery records 1038 of its URIs in huggingface://{repo}/{file} shorthand. A generated unsloth rung whose weights are already shipped in shorthand was therefore not recognised. canonicalURI reduces both spellings to one key and is applied on both sides, taking care that the repo is exactly the first two path segments so sharded quants in a subdirectory still match. A URI in neither form is returned untouched so other hosts dedup on their literal string. Merge also never accounted for entries it had just accepted, so two generated entries sharing a name or a primary URI both landed in add. Several APEX repos share one base model and resolve to the same unsloth counterpart, so the identical rungs are generated twice under the same name. Batch state is tracked locally rather than written back into the caller's ExistingIndex, which a caller may reasonably reuse. Name is still checked before URI: a name collision must block the add regardless of the weights. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): verify variant and tagging invariants in the gallery index Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): scope the apex-entries verifier to what it can actually judge The verifier reported 60 problems against the real gallery, 57 of which were llama.cpp assumptions meeting entries from other backends. A gate that is wrong 57 times out of 60 cannot gate anything. - The weight-count check catches a quant label collision in llama-cpp quant discovery, so it now runs only for overrides.backend: llama-cpp. Entries with no declared backend are skipped because their weights are declared in the referenced url: template, which the verifier never reads. - The dflash/mtp tag check now implements the per-backend table in .agents/adding-gallery-models.md instead of assuming llama.cpp's spec_type: vocabulary. ds4 declares mtp_path:/mtp_draft:; sglang declares speculative_algorithm: in a file this verifier cannot follow, so sglang entries are not judged in either direction. The check stays bidirectional within the backends it does judge. - sha256 is now required on .gguf files only, since every non-GGUF asset in the index belongs to a hand-curated entry outside this generator's scope. Against the current gallery this leaves exactly the three genuine problems: two entries setting spec_type:draft-mtp without the mtp tag, and one entry whose overrides.mmproj names a file it does not download. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): anchor quant matching and invert the sha256 rule UnaccountedQuants matched files to wanted quants with strings.Contains, which reproduces the substring collision it was written to warn about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0 was reported as publishing an unbuilt Q8_0. Subdirectory-sharded UD quants are the normal unsloth layout for large repos, so this fired on realistic input. Match on the quant label as an anchored token instead, the way DiscoverUnslothQuants does, so the diagnostic and the discovery it audits cannot disagree about what a file is. Root-level shards, the layout the diagnostic mainly exists to catch, stay detected. The sha256 requirement was scoped to .gguf, which exempted seven real model weights: wan_2.1_vae.safetensors and clip_vision_h.safetensors across the wan-2.1-*-ggml entries, both load-bearing weights named by gallery/wan-ggml.yaml. Invert the rule so a checksum is required on everything except metadata extensions, which keeps a future weight format covered by default rather than silently exempt. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): wire the apexentries command Adds the generation path to the apexentries command: list the mudler APEX repos, discover each one's quality ladder and its unsloth counterpart's quant rungs from the filenames actually published, render a child entry per build plus a family parent carrying the variants list, dedup against the gallery, and write the additions to -out or append them with -apply. Discovery shortfalls are reported at discovery time rather than left to the verifier. A quant or a tier that discovery drops leaves no trace in a finished gallery file, and because an empty imatrix ladder falls back to the plain one, a repo whose imatrix filenames all fail to match downgrades the whole family silently instead of erroring. Merge's single reused map is split into two reported categories. A URI match means the gallery already ships exactly these weights and referencing the existing entry is correct; a name collision means an unrelated entry owns the name and referencing it would substitute a different build. Multimodal children now declare known_usecases [chat, vision]. An explicit known_usecases suppresses the backend-default fallback, so a chat-only entry carrying an mmproj never matches the vision or multimodal gallery filters. .github/ci is invisible to go list ./..., so a workflow names both generator packages explicitly and their specs finally run on pull requests. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ci): gather APEX builds under the base model entry The hub for a family is the BASE model entry, never a generated *-apex parent. Somebody looking for qwen3.6-35b-a3b has to find every build of those weights under that one name, so a competing qwen3.6-35b-a3b-apex hub would split the family and leave half of it invisible. When the gallery already ships the base entry, a variants block is spliced into it textually, leaving its description, icon, tags, overrides and files untouched. Only a family whose base model the gallery does not ship gets a new hub, still named for the base model and carrying one of the discovered builds as its own payload so it declares a backend the verifier can judge. The line editing is factored into .github/ci/galleryedit, shared with the variantproposals job, so the two cannot drift apart on where a variants block belongs. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(apexentries): treat an unreadable optional counterpart repo as absent HuggingFace answers 401 Unauthorized, not 404, for a repository that does not exist when the request carries no credentials. FetchRepoFiles treated only 404 as absence, so probing for the OPTIONAL unsloth counterpart hard failed for every family that legitimately has none: 27 of the 45 APEX families are community merges that will never have an unsloth build, and a full run failed all of them. Split the fetch so the two call sites can apply different policies to the same response. The APEX repo itself stays strict: a 401 or 403 on a repo the run requires is a real failure and still errors. Only the optional probe tolerates it, because without a token 401 cannot be told apart from absence. That collapse is lossy in one direction, since a private or gated repo also answers 401, so the skipped candidates are named in the run summary alongside the other silent-shortfall counters instead of being dropped in silence. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(apexentries): report full-precision sources as a known exclusion The 45 APEX repos publish their unquantized F16 sources next to the imatrix ladder, flat or sharded. Discovery correctly emits nothing for them, but they were landing in the unclassified total, leaving a permanent baseline of 24 benign lines on every run. That baseline is what the unclassified check exists to prevent: a standing count of known-benign files is exactly what hides the one file that ever genuinely matters. Count full-precision sources separately and give them their own summary line, so unclassified returns to 0 and stays loud when something really is an unknown shape. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(apexentries): namespace local paths by owner and enable MTP builds localPath namespaced downloads by the repo basename alone, so two repos publishing the same filename under different owners collapsed to one local path. LiquidAI/LFM2.5-8B-A1B-GGUF and unsloth/LFM2.5-8B-A1B-GGUF collided that way, and both were offered from the same hub, so installing the second either overwrote the first model's weights or was skipped as already present while recording a sha256 that did not match the bytes on disk. The owner is now its own path segment: owner/repo is globally unique on HuggingFace and neither half can contain a separator, so uniqueness holds by construction. Verify gains a check for the whole class, that no local filename may map to two different upstream URIs. It surfaces seven pre-existing collisions in the gallery, which are left alone here. Entries built from the *-APEX-MTP-GGUF repos now configure MTP rather than shipping the heads inert, matching the pattern the hand-written MTP entries already use: spec_type:draft-mtp with spec_n_max and spec_p_min, tagged mtp, and no draft_model because the heads live in the weights. RenderChild no longer requires a separate drafter file before it will configure a spec type, while the cross-repo drafter path is unchanged. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add the APEX GGUF families as variant ladders Adds the imatrix quality ladder from each mudler/*-APEX-GGUF repo, a fixed subset of unsloth quant rungs where a counterpart repo exists, and the MTP builds, then attaches them to the base model entry so one entry offers every build of the same weights and LocalAI picks the one that fits the hardware. Ten existing base model entries gain a variants list; twenty-seven families that the gallery had no base entry for get one. Builds are discovered from the filenames each repo actually publishes rather than derived from its name, since six repos ship a stem that differs from their repo name. Every file carries a sha256 taken from the HuggingFace API. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )