feat(gallery): fix stale DFlash drafters and add the APEX families as variant ladders#11027
Merged
Conversation
…ame GGUFs The drafters both entries referenced were converted from the pre-merge DFlash PR branch and carry dflash.target_layer_ids. llama.cpp reads dflash.target_layers and refuses the load. The stored values are offset by +1 relative to the HF-side field, so the files cannot be repaired by renaming the key and must be replaced. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The apexentries HuggingFace client was constructed as a raw
&http.Client{Timeout: 60s}. The repo convention (documented in
.golangci.yml, which cannot express this as a forbidigo pattern) is that
all outbound HTTP goes through pkg/httpclient, which refuses redirects by
default and sets a TLS 1.2 floor. The std client follows redirects and
forwards custom credential headers to the redirect target on a cross-host
hop (GHSA-3mj3-57v2-4636). Only a User-Agent is sent today, but this
calls an external API and an HF_TOKEN header added later would leak.
Switch to httpclient.NewWithTimeout, preserving the 60 second timeout.
No behaviour change for the current header set.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
RenderChild left three gaps against the hand-written gallery entries. The generated entries reference gallery/virtual.yaml, which supplies no backend, so every generated entry named no engine at all. All comparable hand-written entries set backend: llama-cpp in overrides; do the same. Set known_usecases to [chat] alongside it: LocalAI falls back to the backend defaults when it is absent, so this is convention rather than breakage, but generated entries should not read differently from their neighbours. The drafter was also assumed to live in the repo publishing the weights. Speculative pairings routinely cross repos, and a drafter URI built from the weights repo 404s at install time. Add ChildInput.DraftRepo, used for both the drafter URI and its local path, falling back to Repo when empty so pairings that do ship the drafter alongside the weights are unchanged. The dflash/mtp tagging rule is untouched: the tag still follows SpecType and nothing else. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Merge exists to stop a second gallery entry being added for weights the gallery already ships, but two gaps let duplicates through on a bulk run. The URI key was compared as an exact string while render.go only ever emits https://huggingface.co/{repo}/resolve/main/{file} and the gallery records 1038 of its URIs in huggingface://{repo}/{file} shorthand. A generated unsloth rung whose weights are already shipped in shorthand was therefore not recognised. canonicalURI reduces both spellings to one key and is applied on both sides, taking care that the repo is exactly the first two path segments so sharded quants in a subdirectory still match. A URI in neither form is returned untouched so other hosts dedup on their literal string. Merge also never accounted for entries it had just accepted, so two generated entries sharing a name or a primary URI both landed in add. Several APEX repos share one base model and resolve to the same unsloth counterpart, so the identical rungs are generated twice under the same name. Batch state is tracked locally rather than written back into the caller's ExistingIndex, which a caller may reasonably reuse. Name is still checked before URI: a name collision must block the add regardless of the weights. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The verifier reported 60 problems against the real gallery, 57 of which were llama.cpp assumptions meeting entries from other backends. A gate that is wrong 57 times out of 60 cannot gate anything. - The weight-count check catches a quant label collision in llama-cpp quant discovery, so it now runs only for overrides.backend: llama-cpp. Entries with no declared backend are skipped because their weights are declared in the referenced url: template, which the verifier never reads. - The dflash/mtp tag check now implements the per-backend table in .agents/adding-gallery-models.md instead of assuming llama.cpp's spec_type: vocabulary. ds4 declares mtp_path:/mtp_draft:; sglang declares speculative_algorithm: in a file this verifier cannot follow, so sglang entries are not judged in either direction. The check stays bidirectional within the backends it does judge. - sha256 is now required on .gguf files only, since every non-GGUF asset in the index belongs to a hand-curated entry outside this generator's scope. Against the current gallery this leaves exactly the three genuine problems: two entries setting spec_type:draft-mtp without the mtp tag, and one entry whose overrides.mmproj names a file it does not download. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
UnaccountedQuants matched files to wanted quants with strings.Contains, which reproduces the substring collision it was written to warn about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0 was reported as publishing an unbuilt Q8_0. Subdirectory-sharded UD quants are the normal unsloth layout for large repos, so this fired on realistic input. Match on the quant label as an anchored token instead, the way DiscoverUnslothQuants does, so the diagnostic and the discovery it audits cannot disagree about what a file is. Root-level shards, the layout the diagnostic mainly exists to catch, stay detected. The sha256 requirement was scoped to .gguf, which exempted seven real model weights: wan_2.1_vae.safetensors and clip_vision_h.safetensors across the wan-2.1-*-ggml entries, both load-bearing weights named by gallery/wan-ggml.yaml. Invert the rule so a checksum is required on everything except metadata extensions, which keeps a future weight format covered by default rather than silently exempt. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds the generation path to the apexentries command: list the mudler APEX repos, discover each one's quality ladder and its unsloth counterpart's quant rungs from the filenames actually published, render a child entry per build plus a family parent carrying the variants list, dedup against the gallery, and write the additions to -out or append them with -apply. Discovery shortfalls are reported at discovery time rather than left to the verifier. A quant or a tier that discovery drops leaves no trace in a finished gallery file, and because an empty imatrix ladder falls back to the plain one, a repo whose imatrix filenames all fail to match downgrades the whole family silently instead of erroring. Merge's single reused map is split into two reported categories. A URI match means the gallery already ships exactly these weights and referencing the existing entry is correct; a name collision means an unrelated entry owns the name and referencing it would substitute a different build. Multimodal children now declare known_usecases [chat, vision]. An explicit known_usecases suppresses the backend-default fallback, so a chat-only entry carrying an mmproj never matches the vision or multimodal gallery filters. .github/ci is invisible to go list ./..., so a workflow names both generator packages explicitly and their specs finally run on pull requests. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The hub for a family is the BASE model entry, never a generated *-apex parent. Somebody looking for qwen3.6-35b-a3b has to find every build of those weights under that one name, so a competing qwen3.6-35b-a3b-apex hub would split the family and leave half of it invisible. When the gallery already ships the base entry, a variants block is spliced into it textually, leaving its description, icon, tags, overrides and files untouched. Only a family whose base model the gallery does not ship gets a new hub, still named for the base model and carrying one of the discovered builds as its own payload so it declares a backend the verifier can judge. The line editing is factored into .github/ci/galleryedit, shared with the variantproposals job, so the two cannot drift apart on where a variants block belongs. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…sent HuggingFace answers 401 Unauthorized, not 404, for a repository that does not exist when the request carries no credentials. FetchRepoFiles treated only 404 as absence, so probing for the OPTIONAL unsloth counterpart hard failed for every family that legitimately has none: 27 of the 45 APEX families are community merges that will never have an unsloth build, and a full run failed all of them. Split the fetch so the two call sites can apply different policies to the same response. The APEX repo itself stays strict: a 401 or 403 on a repo the run requires is a real failure and still errors. Only the optional probe tolerates it, because without a token 401 cannot be told apart from absence. That collapse is lossy in one direction, since a private or gated repo also answers 401, so the skipped candidates are named in the run summary alongside the other silent-shortfall counters instead of being dropped in silence. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The 45 APEX repos publish their unquantized F16 sources next to the imatrix ladder, flat or sharded. Discovery correctly emits nothing for them, but they were landing in the unclassified total, leaving a permanent baseline of 24 benign lines on every run. That baseline is what the unclassified check exists to prevent: a standing count of known-benign files is exactly what hides the one file that ever genuinely matters. Count full-precision sources separately and give them their own summary line, so unclassified returns to 0 and stays loud when something really is an unknown shape. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
localPath namespaced downloads by the repo basename alone, so two repos publishing the same filename under different owners collapsed to one local path. LiquidAI/LFM2.5-8B-A1B-GGUF and unsloth/LFM2.5-8B-A1B-GGUF collided that way, and both were offered from the same hub, so installing the second either overwrote the first model's weights or was skipped as already present while recording a sha256 that did not match the bytes on disk. The owner is now its own path segment: owner/repo is globally unique on HuggingFace and neither half can contain a separator, so uniqueness holds by construction. Verify gains a check for the whole class, that no local filename may map to two different upstream URIs. It surfaces seven pre-existing collisions in the gallery, which are left alone here. Entries built from the *-APEX-MTP-GGUF repos now configure MTP rather than shipping the heads inert, matching the pattern the hand-written MTP entries already use: spec_type:draft-mtp with spec_n_max and spec_p_min, tagged mtp, and no draft_model because the heads live in the weights. RenderChild no longer requires a separate drafter file before it will configure a spec type, while the cross-repo drafter path is unchanged. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds the imatrix quality ladder from each mudler/*-APEX-GGUF repo, a fixed subset of unsloth quant rungs where a counterpart repo exists, and the MTP builds, then attaches them to the base model entry so one entry offers every build of the same weights and LocalAI picks the one that fits the hardware. Ten existing base model entries gain a variants list; twenty-seven families that the gallery had no base entry for get one. Builds are discovered from the filenames each repo actually publishes rather than derived from its name, since six repos ship a stem that differs from their repo name. Every file carries a sha256 taken from the HuggingFace API. Assisted-by: Claude Opus 4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Two things, delivered together.
1. Fixes two broken DFlash entries
qwen3-4b-dflashandqwen3.5-9b-dflashpointed at drafter GGUFs converted from the DFlash pull request branch before it merged into llama.cpp on 2026-06-28. Those files carrydflash.target_layer_ids, while llama.cpp readsdflash.target_layers, so the model fails to load with:The stored values are also offset by +1 relative to the HuggingFace-side field, so the files cannot be repaired by renaming the key. Both entries now point at post-rename drafters, verified by reading the GGUF headers directly.
The other three DFlash entries were checked and are correct as they stand.
2. Adds the APEX families as variant ladders
A new generator at
.github/ci/apexentriesreads the HuggingFace API and produces gallery entries for the 45mudler/*-APEX-GGUFrepos: the imatrix quality ladder, a fixed subset of unsloth quant rungs where a counterpart repo exists, and the MTP builds.Everything attaches to the base model entry, so a user looking for
qwen3.6-35b-a3bgets one entry offering every build of those weights and LocalAI picks the one that fits the hardware. Ten existing base entries gain avariants:list; 27 families the gallery had no base entry for get one.gallery/index.yamlgoes from 1274 to 1527 entries: 6037 insertions, zero deletions. The ten splices are pure insertions, so no anchor, merge key, or surrounding YAML was disturbed.Design rules the generator follows
gemma-4-26B-A4B-it-APEX-GGUFpublishesgemma-4-26B-A4B-APEX-*.gguf, others drop a-2603suffix or anNVIDIA-prefix.dflash/mtptag is applied if and only if the entry itself setsspec_type:. Variant ranking reads tags and nothing else, so a tag without the configuration promotes a build that is not faster, and configuration without the tag leaves a faster build ranked as plain..agents/adding-gallery-models.mdis the authority.sha256always comes from the APIlfs.sha256. A missing hash is fatal rather than guessed, because on Xet-backed repos the superficially similar field is a Xet hash, not the content hash. One repo trips this and is left failing rather than papered over..github/ci/galleryedit, now shared with the existingvariantproposalstool.Verification
go run ./.github/ci/apexentries -verify gallery/index.yamlreports only pre-existing problems, line-for-line identical to a pre-apply runmodel:at the00001-of-shard with every shard listedgo test ./core/gallery/...green;apexentriesandvariantproposalssuites green.github/workflows/ci-tools-tests.yamladded, because.github/ci/is invisible togo list ./...so these tests never ran in CIPre-existing issues surfaced but deliberately not fixed
The new verifier found these in the current gallery. They predate this branch and are left for a separate change:
qwythos-9b-v2andqwen3.6-27b-fable-fusion-711-...setspec_type:draft-mtpbut carry nomtptag, so they rank as plain buildsqwen3-vl-reranker-2b-i1configures the 2B projector while downloading the 8B onewan-2.1-*files, a VAE and a CLIP vision encoder, havesha256: ""and are therefore unverified downloadsmmproj-F32.ggufbetweenqwen3.5-9bandqwen3.5-397b-a17bKnown limitation
The generator cannot distinguish a transient HuggingFace response from a counterpart repo that does not exist, because HuggingFace answers 401 rather than 404 for a nonexistent repo to an unauthenticated client. 135 live probes showed no flakes and every stable counterpart is represented in this run, but a future run wants a retry or an
HF_TOKENfor truthful 404s.Assisted-by: Claude Opus 4.8 [Claude Code]