AutoTuner for llama.cpp

Interactive launcher for llama-server that detects your hardware, scans your local GGUF collection, and auto-tunes context length, KV-cache quantization, GPU offload, threading, and batch size to fit in the RAM/VRAM you actually have free — without manual edits.

GUI-Design

Features

• Interactive terminal menu — pick from whatever GGUFs are in your models folder, no editing required.

• End-to-end OCR documents (GUI + TUI) — OCR models get a 📄 action that accepts images, multi-frame TIFF/GIF, PDF, Word/Office/OpenDocument, presentation, spreadsheet, or whole-folder input. Office files are converted with LibreOffice before the model claims RAM; PDFs are rendered with PyMuPDF; pages are normalized with Pillow and sent through llama-server's multimodal API. Per-page files, a combined Markdown/text result, and a hashed JSON manifest go to the selected output folder. The source documents are never modified.

• Visible throughput logs — AutoTuner emits --perf on current llama.cpp builds so the separate terminal keeps showing prompt/eval timings and tokens/s; append --no-perf if you want quieter logs.

• Hardware auto-detection — works on AMD (ROCm), NVIDIA, Intel, and Apple Silicon (unified memory). Multi-GPU is supported via automatic --tensor-split. The split strategy depends on the model type:

  • Dense models use a priority-weighted split: a model that fits the largest card is pinned to it (the second GPU stays free for gaming/OBS); larger models put the bulk of the weights on the high-priority card.
  • MoE models that don't fit the primary card alone use a capacity-fill split instead — both GPUs are packed to roughly the same utilisation so the maximum number of expert layers stays resident in VRAM (every expert that lands on the GPU instead of spilling to CPU via --n-cpu-moe is a real speed win). This replaces the old priority-weighted behaviour for MoE, which stranded several GB on the secondary card and slowed the model down.

  Device visibility is pinned via HIP_VISIBLE_DEVICES and GGML_VK_VISIBLE_DEVICES so it works on both ROCm and Vulkan builds.

• Free-memory aware — context length and KV quant are picked to use the RAM/VRAM that's actually free right now, not a hard-coded cap. The original v1 cap of 16k context is gone.

• Per-family YAML profiles in settings/ — override sampling, max context, chat template, and llama-server flags per model family. Easy for contributors to extend without touching Python.

• Companion-file auto-pairing — sibling files don't pollute the model menu, they're attached to their main model:

  • mmproj projectors → vision (longest-prefix wins). The mmproj marker is detected anywhere in the filename, not just as a leading mmproj- prefix — so a projector named with the marker mid-name (e.g. qwen3.6-35b-a3b-mxfp4-moe-mmproj-f16.gguf, where the vendor put the quant label before mmproj) is now paired correctly. Projectors saved with a literal .mmproj extension (some audio projectors) are picked up too, even though they don't match the *.gguf glob. Matching is separator-tolerant (-moe vs _moe no longer blocks a pair) but still size-specific, so a 2B model never grabs a 0.8B projector. When a model ships several projector precisions side by side (…-bf16, …-f16, …-f32), all are kept as candidates and a dropdown in the Launch options lets you switch between them. The auto pick prefers the highest precision (f32 > f16 > bf16) on an otherwise-equal name match; your manual choice is remembered per model.
  • *-assistant-*.gguf / *-draft-*.gguf / mtp-*.gguf / dspark-*.gguf → speculative decoding (smallest matching sibling wins). A leading mtp- explicitly marks an external draft head even when newer Qwen-based heads are several GiB; large infix …-MTP-… target models remain normal runnable models. EAGLE-3, DFlash, and b10329 DSpark sidecars are also classified from GGUF architecture/tensor metadata and receive their required --spec-type.

• Themes and appearance editor — choose built-in System, Dark, Dark Gray, Light, High Contrast, or the pink-accented Midnight Rose theme in ⚙ Settings, or copy one in the in-app color/font editor and save it as a safe JSON user theme. See the complete theme guide.

• Favorite models stay at the top — click the star left of a model name to mark it. Active and inactive star colors follow the selected theme, and the choice is persisted across AutoTuner restarts.

• Persistent folder tree — folders and the Favorites section start open, but every branch you manually open or close keeps that exact state when you switch list/folder views and after restarting AutoTuner.

• Capability badges in the model list — symbols make it obvious what each model can do at a glance:

  • 👁 vision (mmproj projector paired)
  • ⚡ draft (assistant sibling for speculative decoding)
  • 🧠 thinking (chat template emits <think> / reasoning_content)
  • 🛠 tool-use (chat template advertises tool_calls / function_call)
  • 📄 OCR/document parsing (known OCR architecture/name)

  Detection reads the GGUF chat template directly — no name-based guessing — so Qwen3-Coder (no thinking) and Qwen3-Embedding (neither thinking nor tools) are correctly excluded.

• Model actions on right-click — choose GGUF-Ordner öffnen to open the containing directory, or use Expert Settings kopieren/einfügen to transfer the active performance mode's complete Expert snapshot to another model. The internal clipboard retains the source mode (safe, balanced, throughput, or low_vram) so settings cannot silently land in a different mode.

• Reads GGUF metadata — pulls n_layers and context_length straight from the file so partial GPU offload (-ngl) is exact.

• Author-recommended samplers from GGUF metadata — many models embed their recommended sampler defaults in general.sampling.* (e.g. Qwen3.5/3.6 ship temp 1.0 / top_k 20 / top_p 0.95). The tuner now reads these and uses them to fill any sampling value a matched YAML profile leaves unspecified. Priority per field is: a matched family profile's explicit value wins first (these are hand-tuned), then the GGUF recommendation, then the generic default. The practical effect: a model with no tailored profile (so it would otherwise fall back to the generic temp 0.7 / top_k 40) now runs on its intended samplers — a frequent cause of repetition loops and broken tool-calls on models tuned for a low top_k with a non-zero min_p.

• Measured performance profiles for every mode — select a normal text/chat GGUF and click 🚀 Performance test. The setup dialog can test any or all of safe, balanced, throughput, and low_vram, optionally enable YaRN, tune MTP/draft n-max, or queue every benchmarkable scanned model. Each mode gets its own persistent Expert snapshot. Choose Quick for a deterministic prompt covering 12% of context or Normal for 25% (both bounded at 65,536 tokens); every candidate starts a fresh private llama-server, uses an excluded warm-up, and requests 256 n_decode tokens. The 📊 Performance analysis button opens separate Quick and Normal tiles that list every tested model/mode, graph native prompt-processing, n_decode, and measured end-to-end throughput, and explain exactly how each metric is collected. The two workloads have independent storage and chart scales, so their results are never mixed. Winner ranking uses real workload time rather than the old geometric score that could exaggerate prompt-only gains. The best mode for each model is marked, remembered, and automatically selected. A requested context above the conservative static estimate can be tried in the isolated server; only a successful load + inference run is saved (for example, a backend-proven 110,592-token profile is no longer rejected merely because static headroom estimated ~78k). low_vram remains the explicit, safe way to place KV in abundant system RAM via --no-kv-offload; spare RAM is never added to a full-offload VRAM budget implicitly. Settings → Performance profiles exports/imports a portable JSON backup matched by GGUF filename and byte size. AutoTuner never changes clocks, voltage, fan curves, or power limits.

• Multi-server (run several models at once) — Launch no longer refuses while a server is running. Each new model gets the next free port: 0 servers → 1234, 1 → 1235, 2 → 1236, … When a server is stopped or exits, its port is reclaimed and the counter effectively resets (the next launch reuses the freed port). The status bar shows how many servers are live; Stop stops the most recent one.

• GPU load-balancing for the 2nd/3rd model — before launching an additional model the AutoTuner re-reads live VRAM (so it sees what the already-running models actually hold) and steers the new model onto the emptier card via HIP_VISIBLE_DEVICES / GGML_VK_VISIBLE_DEVICES. If no GPU has room it refuses with a clear message instead of piling everything onto a card that's already full (e.g. an R9700 sitting at 31/32 GB). The first model still uses the normal automatic multi-GPU split.

• Host-memory prompt caching (--cache-ram / -cram) — auto-enabled with a Launch-options toggle to turn it off. Cached prompt prefixes live in system RAM and are hot-swapped back when a new request shares a long prefix (system prompt, RAG scaffold, Roo-Code preamble), collapsing time-to-first-token on repeated prompts. Current llama.cpp builds (b10045+) support this for Vision/mtmd too; older or unprobeable builds conservatively receive --cache-ram 0 when Vision is active. The default cap is 2048 MiB, editable in the GUI or with --cache-ram-mib; the reserved amount is included in the RAM estimate.

• Sticky GUI choices — the Qt launcher remembers each model's selected mmproj and draft/MTP dropdown entries plus thinking / n-gram / prompt-cache toggles in autotuner_settings.json. n-gram self-speculation now defaults on for every model; an explicit per-model off choice remains authoritative. Each model also remembers its selected/fastest performance target and independent Expert settings for all four targets. Switch models, restart, or change targets — the corresponding choices return.

• Fork-folder memory — if you point the GUI at a parent folder that holds several *_llama.cpp builds (e.g. C:\LAB\ai-local), the next launch re-expands the same set of builds in the dropdown. Both build-number names such as b10566_llama.cpp and semantic-release names such as v0.2.0_llama.cpp are recognized. No more re-navigating one folder up after every restart.

• Window geometry, state & inner layout — QMainWindow saveGeometry() (size, position, maximize-state) and saveState() (toolbars/docks) are persisted as base64 in the settings JSON. In addition, each inner QSplitter saves its own handle positions under a stable object name, so the arrangement of the panes inside the window (model-list vs config width, and the log-panel height) is restored too — not just the outer window size. saveState() alone does not round-trip plain central-widget splitters, hence the separate per-splitter persistence.

• Global font size — persistent font size (clamped 7..22), applied immediately on app start (no flash of the default).

• Application settings & Info — the ⚙ Settings toolbar button and the Windows title-bar system menu (click the top-left app symbol) provide Settings plus About AutoTuner with the visible version and GitHub link. The window title also always shows the running version. Settings controls per-user login autostart on Windows, Linux, and macOS, plus optional X → notification area/system tray behavior. Both options are disabled by default; the tray menu and the dedicated Quit button always exit normally.

• Stable theme preview — switching or previewing a theme preserves the current window and pane/splitter arrangement; responsive Appearance controls retain full theme/font names through sensible widths and tooltips.

• Reasoning effort — selectable per model: auto / off / minimal / low / medium / high / extra_high. Think-budget (spin-box, -1 = off, 0 = stop immediately, N = token budget) and optional preserved reasoning history (--reasoning-preserve) live in the Expert panel.

Vision control

You can disable vision (mmproj) support in two ways:

1. Command-line flag:

   python auto_tuner.py --model "Qwen3.6" --novision

2. GUI checkbox in Launch options. A second checkbox can keep an active projector in system RAM (--no-mmproj-offload); the memory preview then moves its footprint from VRAM to RAM. When the model ships several projector precisions (…-bf16 / …-f16 / …-f32), an mmproj dropdown appears above the checkboxes so you can pick which one to load. The chosen projector is remembered per model in autotuner_settings.json under mmproj_selection. The auto pick defaults to the highest precision available.

Note on prompt caching + vision: llama.cpp b10045+ can cache multimodal prompts. AutoTuner enables it on those builds; older or unprobeable binaries safely fall back to --cache-ram 0 while Vision is active. This was runtime-verified on b10058 with a repeated Gemma 4 image request (cached_tokens: 0 → 279).

OCR documents

Select an OCR model with its matching mmproj (for example Unlimited-OCR-BF16.gguf + mmproj-Unlimited-OCR-F16.gguf). In the Qt GUI, click 📄 OCR… or double-click the OCR model. Choose one or more files/folders, an output folder, PDF page range, DPI, output format, and token budget. The GUI converts/prepares documents first, loads the model, verifies the server's model alias, shows page progress, and optionally releases the server/VRAM afterward.

Terminal equivalent:

python auto_tuner.py \
  --model "Unlimited-OCR" \
  --ocr-input document.pdf \
  --ocr-input scans/ \
  --ocr-output ocr-results \
  --ocr-pages 1-5 \
  --ocr-dpi 220 \
  --ocr-max-tokens 4096 \
  --non-interactive --yes

PDF rendering requires PyMuPDF and image normalization requires Pillow (both are installed by requirements.txt). Word/Office/OpenDocument input additionally requires a local LibreOffice installation; without it AutoTuner gives a clear error and leaves the source untouched. Unlimited-OCR uses its verified document parsing. prompt, F16 KV, deterministic sampling, explicit Flash Attention off, and requires llama.cpp b10287+ (a current build such as b10362 is recommended). There is no CMake/build flag for max_tiles=32. The value comes from the projector GGUF metadata. For the full 32-tile Unlimited-OCR path, the mmproj must contain clip.vision.preproc_max_tiles=32; AutoTuner warns when an older projector would silently fall back to 9 tiles. Reconvert Unlimited-OCR's mmproj with a current convert_hf_to_gguf.py --mmproj (the current converter writes 32 automatically), or download a newly converted projector. Rebuilding llama-server cannot repair an old mmproj.

AutoTuner's OCR workflow uses the persistent OpenAI-compatible HTTP server, not llama-mtmd-cli. Therefore a llama.cpp directory is shown in the fork picker only after it contains build/bin/Release/llama-server.exe (or the native Linux/macOS binary). building llama.cpp/ocr_llama_build.txt fixes the legacy PR #17400 recipe by building both llama-server and llama-mtmd-cli; that old fork remains for DeepSeek-OCR v1. Unlimited-OCR should use the current mainline build from building llama.cpp/llama_build.txt.

Installation

git clone https://github.com/DaWasteh/Auto-Tuner.git
cd "Auto-Tuner"
pip install -r requirements.txt

You also need a working llama-server binary. The tuner automatically discovers binaries in common local setups (like C:\LAB\ai-local\), or you can specify one via --server.

Update (GUI-Button oder Terminal)

In der Qt-GUI gibt es oben in der Toolbar den Button ⬆ Update. Die laufende Version steht im Fenstertitel sowie in den Update-Dialogen und -Meldungen. In einem Source-Start prüft er den Git-Branch beziehungsweise das GitHub-Source-ZIP, nicht Release-Assets. In einem echten git clone nutzt er git pull --ff-only; in einem heruntergeladenen ZIP/Release-Ordner ohne .git lädt er automatisch das aktuelle GitHub-Source-ZIP herunter und spielt es über die bestehende Installation. Bei geänderter requirements.txt führt er automatisch pip install -r requirements.txt aus. Vor dem Update wird autotuner_settings.json gesichert und danach wiederhergestellt — deine lokalen Pfade, Ports, Overrides und UI-Einstellungen werden also nicht überschrieben (auch bei älteren Klonen, in denen die Datei noch versehentlich von Git getrackt wird). Nach einem erfolgreichen Update AutoTuner neu starten.

Danke für die Update-Button-Idee an nextscript.

Wenn du AutoTuner per git clone installiert hast, kannst du neue Versionen weiterhin über das Terminal holen. Deine persönlichen Einstellungen bleiben auch dabei erhalten, solange autotuner_settings.json lokal ignoriert ist.

1. In den App-Ordner wechseln (dorthin, wo du geklont hast):

cd "H:/GitHub/Auto Tuner"

2. Aktuelle Änderungen herunterladen:

git pull

Falls du lokal an den Python-Dateien herumgepfuscht hast und git pull mit einem Konflikt abbricht, kannst du deine lokalen Änderungen verwerfen und den Upstream-Stand übernehmen:

git stash            # lokale Änderungen kurz beiseite legen
git pull
git stash drop       # beiseite gelegte Änderungen verwerfen
# oder: git stash pop  → Änderungen wiederherstellen (evtl. Konflikte lösen)

⚠️ In aktuellen Klonen ist autotuner_settings.json gitignored; git stash berührt sie dann nicht. Wenn dein alter Klon die Datei noch als geändert anzeigt, nutze bevorzugt den ⬆ Update-Button — er sichert und restored die Datei explizit.

3. Abhängigkeiten aktualisieren (nur nötig, wenn sich requirements.txt geändert hat — schadet aber nie):

pip install -r requirements.txt

Falls du eine virtuelle Umgebung nutzt, aktiviere sie vorher. Projekt-Konvention: Windows nutzt .venv, Ubuntu/Linux nutzt .venv_linux.

# Ubuntu/Linux
python3 -m venv .venv_linux
.venv_linux/bin/python -m pip install -r requirements.txt
source .venv_linux/bin/activate

REM Windows
py -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

4. App starten — danach einfach wie gewohnt:

python qt_launcher.py        # GUI
# oder
python auto_tuner.py         # Terminal

Kurzform (wenn sich nichts an den Abhängigkeiten geändert hat):

git pull && python qt_launcher.py

Compiled build (.exe / Linux binary)

Für Einsteiger gibt es kompilierte noconsole-Versionen ohne separate Python-Installation: Windows 10/11 x64 (.exe), Linux x64 und macOS Apple Silicon arm64. Der Linux-Build wird bei jedem Release automatisiert auf Ubuntu, Fedora, Arch Linux, Linux Mint, CachyOS, Kali Linux und Debian gestartet; Windows und macOS erhalten zusätzlich einen nativen Frozen-Binary-Smoke-Test. Alle drei Builds verwenden denselben ⬆ Update-Button.

Der macOS-Build ist derzeit nicht mit einer kostenpflichtigen Apple Developer ID signiert/notarisiert. Gatekeeper kann deshalb beim ersten Start warnen: App im Finder mit Rechtsklick → Öffnen starten und die Rückfrage bestätigen; falls nötig unter Systemeinstellungen → Datenschutz & Sicherheit → Dennoch öffnen freigeben. Nur das Asset aus dem offiziellen AutoTuner-GitHub-Release verwenden. Intel-Macs und ARM-Linux sind derzeit Source-Installationen, keine bereitgestellten Frozen-Assets.

Bauen

PyInstaller kann nicht cross-kompilieren — die .exe wird auf Windows, der Linux-Binary auf Linux und der macOS-Build auf macOS erzeugt. Alle landen als Assets im selben GitHub-Release. GitHub Actions prüft den Quellcode zusätzlich in nativen Containern der oben genannten Linux-Distributionen.

# einmalig im Build-Environment:
python -m pip install pyinstaller
python -m pip install -r requirements.txt

# dann das jeweilige Binary bauen:
python build_exe.py
# → Windows: dist/AutoTuner.exe   |   Linux: dist/AutoTuner-Linux

build_exe.py bündelt settings/*.yaml und das App-Icon aus assets/ als read-only Daten mit; unter Windows wird assets/AutoTuner.ico zusätzlich in die .exe eingebettet. Nutzer-State (autotuner_settings.json, Logs) liegt persistent neben dem Binary und bleibt bei Updates erhalten.

Release / Auto-Update

1. autotuner_version.py → VERSION hochzählen (z. B. "1.1.0").
2. Commit + Tag v1.1.0 (das Tag-Format v<VERSION> ist Pflicht — der Updater streift das führende v vor dem Vergleich).
3. Der Release-Workflow erstellt die drei ZIP-Assets für Windows, Linux und macOS. Veröffentlicht wird erst, nachdem das Linux-Binary die Smoke-Tests auf allen unterstützten Distributionen bestanden hat.
4. Der Update-Button in der kompilierten Version holt sich automatisch das zum laufenden OS passende Asset, lädt es herunter und tauscht das Binary über einen Swap-Shim nach Neustart aus (Windows sperrt die laufende .exe, deshalb der Shim; auf Linux geht das direkte mv über das laufende ELF).

Source-Installationen (Entwickler) nutzen weiterhin den git/ZIP-Updater; die kompilierte Version wählt automatisch den Binary-Swap-Pfad (sys.frozen).

Usage

Point it at a folder of *.gguf models — it will recurse:

python auto_tuner.py --models-path /path/to/models

Or set the environment variable once:

export AUTOTUNER_MODELS=/path/to/models     # Linux / macOS
setx  AUTOTUNER_MODELS  D:\models           # Windows
python auto_tuner.py

Pick a model from the menu. Once it's running, point your client at:

http://127.0.0.1:1234

Works with the built-in llama.cpp Web UI, VS Code extensions like Continue / Cline, Open WebUI, or any OpenAI-API client.

Terminal TUI

python auto_tuner.py

The dependency-free terminal interface is designed for enthusiasts who prefer a fast keyboard workflow. It shows the shared AutoTuner version, responsive model cards, capability badges, the active performance target/GPU pin, and a compact memory/config summary. ANSI color is used only in compatible TTYs; logs, pipes, and --plain automatically use an ASCII-only, ANSI-free view.

For repeatable scripts, use an unambiguous model with --non-interactive (or --dry-run / --yes). These paths do not read prompts: available vision, draft, thinking, and n-gram features use their normal defaults. n-gram is on by default for every model; use --no-ngram to disable it (--ngram remains a compatible explicit-on spelling). The terminal UI shares the launch engine, but the GUI remains the place for persisted per-model Expert controls; advanced llama-server options can still be passed after --. OCR models expose the same shared document pipeline through --ocr-input / --ocr-output; the job-owned server is stopped automatically when processing finishes.

Qt GUI

python qt_launcher.py

Same engine as the terminal launcher, plus a few quality-of-life bits that only make sense with persistent state:

• OCR dialog and progress. OCR models show 📄 OCR…; double-clicking one opens the same workflow as the TUI. Inputs are prepared off the GUI thread, mutable launch controls are locked until the validated server starts, Cancel interrupts LibreOffice/HTTP/model loading, and successful output can be opened directly from the completion dialog.
• Sticky per-model options. Choose mmproj and draft/MTP directly from their dropdowns (none disables either feature), then toggle thinking / n-gram / prompt-cache as needed. n-gram starts on but can be disabled per model. Choices survive model switches and restarts.
• Expert settings are saved per model and performance target (autosave). Auto-cascade pins and full Manual setups are independently persisted for safe, balanced, throughput, and low_vram; changing the toolbar target restores that target's values instead of leaking one winner into every mode. Auto-mode saves re-cascade against current hardware. A context proven by the isolated real benchmark is stored as an exact Manual profile so a later static estimate cannot silently clamp it. ⟲ Reset resets only the current mode. Settings → Performance profiles exports/imports these snapshots and their measurement evidence without copying unrelated application settings.
• Run several models at once. Launch stays enabled while servers are running — each new model is placed on the emptier GPU and given the next free port (1234, 1235, 1236, …). Stop a server and its port is freed for the next launch. If no card has room, the GUI tells you instead of overcommitting a full GPU.
• GPU pin dropdown (toolbar → GPU). Click-path equivalent of the CLI --gpu flag: choose Auto for the usual free-VRAM-aware selection, or pick a card by name to hard-pin the next launch to it and hide the others. The card list is filled from detected hardware, so it shows your actual GPUs (e.g. R9700, 9070) rather than fixed labels. Use it to force a second server onto the still-empty card instead of letting it pile onto the busy one. The choice is persisted as forced_gpu in autotuner_settings.json and feeds the same compute_config(force_gpu=…) path as the CLI flag, so the config preview updates the instant you change it.
• Fork picker remembers the parent folder. Hit 📂 Fork and pick a directory that holds multiple *_llama.cpp builds — every build appears in the dropdown next time too, not just the last one you used. The active build within that container is also restored.
• Live config preview. The right pane recomputes context / KV / placement whenever you tick a checkbox or change the performance target — no need to launch first.
• Honest load status (/health handshake). After launch the status bar shows Loading model and only flips to Ready once the server's GET /health returns 200. Big MoE models can take a while to load (or fail mid graph-build) — the GUI no longer claims "Running" the instant the PID exists. A crash during load is surfaced as Server exited.
• Window geometry + inner layout persistence. Window size, position, maximize-state and toolbar status are saved, and the inner pane arrangement (model-list vs config width, log-panel height) is restored via each splitter's own saveState() — not just the outer window.
• Font persistence. The global QApplication font size is persisted and applied immediately on start (_change_font). No more flash of the default font size.
• Reasoning panel (Expert panel). New section with:
  • Dropdown "Effort": auto / off / minimal / low / medium / high / extra_high
  • SpinBox "Think budget": -1 = off, 0 = stop immediately, N = token budget
  • Checkbox "Preserve reasoning history" → --reasoning-preserve The values are translated into --reasoning, --reasoning-budget, --chat-template-kwargs and --reasoning-preserve in cfg.extra_cli_flags.

Useful flags

Flag	Description
--models-path PATH	Folder to scan (default ./models, env AUTOTUNER_MODELS)
--settings-path PATH	Folder with YAML profiles (default ./settings)
--server PATH	Path to llama-server (default looks on $PATH, env LLAMA_SERVER)
--host HOST	Bind address (default 127.0.0.1)
--port N	Server port (default 1234). In the GUI this is the base port; each additional concurrent server gets the next free one (1235, 1236, …)
--ctx N	Override the auto-tuned context length
--model SUBSTR	Skip the menu, pick a model by name substring
--gpu NAME	Hard-pin the server to a single GPU by name substring (e.g. --gpu 9070, --gpu R9700). Overrides the persisted forced_gpu; omit for free-VRAM-aware auto selection. The GUI exposes the same pin via the toolbar GPU dropdown
--ngram / --no-ngram	Explicitly enable / disable default-on n-gram self-speculative decoding
--no-mmproj-offload	Keep an active vision projector in system RAM instead of VRAM; its size is moved into the RAM budget
--cache-ram-mib MIB	Bound host prompt caching (-1 unlimited, 0 disabled); without this flag the saved GUI cache limit is used
--no-prompt-cache	Disable host-memory prompt caching (--cache-ram 0). Caching is auto-on; Vision requires llama.cpp b10045+ and falls back to off on older/unprobeable builds
--ocr-input PATH	Run OCR for a file or recursively for a folder; repeat for several inputs
--ocr-output DIR	Parent folder for the timestamped OCR job/results
--ocr-pages RANGE	PDF/multi-frame page selection such as 1-3,5 (default all)
--ocr-dpi N	PDF render DPI, safely clamped to 72–600 (default 220)
--ocr-max-tokens N	Maximum generated tokens per page (default 4096)
--ocr-format {markdown,text}	Per-page and combined output format
--ocr-keep-rendered	Keep normalized/rendered PNG pages for inspection
--ocr-keep-grounding	Preserve OCR bounding-box/grounding markup instead of cleaning it
--dry-run	Print the command and OCR plan, don't start the server; never opens TUI prompts when used with --model
--yes / -y	Skip the launch confirmation prompt and use non-interactive defaults with --model
--non-interactive	Require --model and never read prompts; intended for scripts/CI
--mode {chat,coding}	Select profile sampling mode (default: saved GUI preference or chat)
--plain, --no-color	Use ASCII-only output, or retain Unicode while disabling ANSI color
--debug, --debug-category NAME	Enable all debug output or one of hardware, scanner, llama_cpp, config
--force-mlock	Force the non-mmap locking path when resources/OS permissions allow it (Expert GUI exposes all --load-mode choices)
--performance-target {safe,balanced,throughput,low_vram}	VRAM utilisation preset (see below)
-- <args...>	Anything after -- is forwarded to llama-server

Performance targets (--performance-target)

A single switch that controls how aggressively the AutoTuner reserves VRAM. It changes both the safety bands and the KV-cache budget that gets reserved up front during MoE layer placement, so picking the right tier can move several expert layers between GPU and CPU.

Tier	KV reservation (MoE / dense)	VRAM safety	When to use
safe	128 k / 64 k tokens	0.30 GB	Long-context sessions (>64 k), maximum stability
balanced (default)	64 k / 32 k tokens	0.25 GB	General use — moderate optimisation that helps everyone
throughput	32 k / 16 k tokens	0.15 GB	Short-context inference (chat, reasoning ≤32 k); pushes more expert layers onto the GPU for higher tokens/s
low_vram	KV → system RAM	0.15 GB	Dedicated low-VRAM / high-RAM boxes (e.g. 8 GB VRAM, 64 GB RAM). Emits --no-kv-offload; attention follows the KV to the CPU. On Apple/iGPU unified memory this changes compute placement but does not increase physical capacity.

Why low_vram exists

On a GPU too small to hold both a MoE model's expert weights and its KV cache, the leftover VRAM after expert placement throttles context to a few thousand tokens — useless for agentic coding, which typically needs 90–130 k. The other tiers keep the MoE KV cache in VRAM (the Vulkan backend asserts when MoE KV spills to RAM mid-split), so they cannot help here. low_vram sidesteps the whole problem by telling llama.cpp to keep the entire KV cache in system RAM (--no-kv-offload); the model's 64 GB of RAM becomes the context budget instead of the ~1 GB of leftover VRAM. Experts still run on the GPU where they fit (--n-cpu-moe), so only attention is paid for in speed. Opt-in only — safe/balanced/throughput are completely unaffected.

Resolution priority (highest wins): explicit CLI flag → GUI dropdown → performance_target: in the model's YAML profile → balanced default. Unknown values are silently ignored, so a typo in a YAML never breaks anything.

A profile can declare its preferred tier in YAML:

# settings/qwen3_5-3_6.yaml
performance_target: throughput   # MoE — wants every spare GB on the GPU

The user choice (CLI / GUI) always wins over the profile recommendation.

Model loading and memory locking (--load-mode)

The Expert panel exposes llama.cpp's complete model-loading strategy: auto (upstream policy; b10364 avoids mmap on iGPUs), none, mmap, mlock, mmap+mlock, and dio. Starting with b10151, mlock means lock normally-read model memory without mmap, while mmap+mlock explicitly combines mapping and locking. AutoTuner emits the non-deprecated --load-mode MODE form and migrates old per-model mlock/no_mmap checkbox snapshots automatically.

The automatic tuner still enables locking only when available RAM/VRAM and OS permissions make it safe. Old or unprobeable GPU builds retain the conservative Vulkan host-buffer crash guard; versioned b10151+ builds allow the Expert locking choices.

Automatic behavior:

Scenario	Condition	Result
GPU backend present	Any detected GPU; automatic config has no resolved binary yet	Locking stays off conservatively; Expert mode permits it after verifying b10151+
CPU-only model	total_ram > 32 GB AND free_ram > model_ram_on_cpu + 8 GB	non-mmap mlock selected when OS permissions allow it; on b10107–b10150 binaries non-mmap locking is unavailable and silently drops to mmap
Insufficient memory	Safety reserve not met	Disabled (fallback to default mmap)

Force memory locking:

Use --force-mlock to override the automatic decision and always enable memory locking when the OS permits it:

python auto_tuner.py --force-mlock

This is useful when you know your system has enough memory but the tuner's conservative thresholds would otherwise skip it.

Debug output:

The tuner prints the mlock decision before every launch:

  [mlock] decision: model=Qwen3.6-35B-A3B-UD-Q6_K
         full_offload=True  vram=18.5GB  ram=0.0GB
         sys: total_vram=24.0GB  free_vram=5.2GB  total_ram=32.0GB  free_ram=12.1GB
         force_mlock=False  -> mlock=True  no_mmap=True

Environment variables

Variable	Default	Purpose
AUTOTUNER_MODELS	./models	Where to scan for *.gguf files
LLAMA_SERVER	llama-server	Path or name of the server binary
LLAMA_CPP_DIR	(auto-detected)	Your llama.cpp checkout. If set, the auto-tuner will look for build/bin/[Release/]llama-server[.exe] inside it.

Server binary auto-discovery

The tuner automatically searches for binaries in common local layouts. If you have a workspace like this, it "Just Works" without any flags:

L:\GitHub\
└── Auto Tuner\         ← clone of this repo
L:\LAB\
└── ai-local\
    ├── llama.cpp\      ← standard build
    ├── tq_llama.cpp\   ← Turbo-Quant build
    ├── ik_llama.cpp\   ← Gemma 4 external drafter (fork still needed)
    └── 1b_llama.cpp\   ← BitNet fork (Ternary-Bonsai)
I:\
└── models\             ← your models

It looks for llama-server inside these directories (including build/bin/... subpaths).

Quantization Modes

When you start the tuner, you can choose between:

1. Standard-Quant: Uses standard llama.cpp binaries.
2. Turbo-Quant: Uses the tq_llama.cpp binary for faster inference.

KV precision and TurboQuant options

Auto mode first preserves the requested/native context window, then spends any remaining memory on the least-quantized KV cache that fits. Its quality ladder starts at F16 (or profile-requested BF16), then steps through asymmetric F16/Q8, Q8, Q5, and Q4 pairs. On multi-GPU systems, a peer GPU is used when it is needed to preserve context or improve KV precision; it stays free only when the primary card already fits both goals.

The Expert K/V dropdowns expose the mainline types f16, bf16, q8_0, q5_0, q5_1, q4_0, q4_1, and iq4_nl, plus fork-only turbo2, turbo3, and turbo4. Turbo KV is no longer duplicated as a Launch options checkbox. Selecting a TurboQuant type shows a special-fork warning with Dismiss and Never Show Again; current mainline llama.cpp rejects those types.

kv_quant_factor() also understands tq3_0, turbo3_tcq, and the normal to Turbo mapping used by compatible terminal/fork flows:

Label	Turbo-Quant	Factor (vs F16)
q8_0	turbo4	~3.8x
q5_0	turbo3	~4.3x
q4_0	turbo3	~4.3x

Specialized Binary Logic

The tuner intelligently selects the best binary based on your model and settings:

• Gemma 4 (with external draft) $\rightarrow$ uses ik_llama.cpp (external sibling drafter still requires the fork).
• Gemma 4 (without draft) $\rightarrow$ uses standard llama.cpp.
• Integrated MTP (e.g. Qwen3.6-27B-MTP) $\rightarrow$ uses standard llama.cpp (native since b9190+; PR #22673 in mainline since 16 May 2026; no fork needed).
• Ternary-Bonsai $\rightarrow$ uses 1b_llama.cpp.
• Turbo-Quant Mode $\rightarrow$ uses tq_llama.cpp.

Example — run Devstral, override context, and pass an extra flag (--metrics is enabled by default, so this just shows pass-through):

python auto_tuner.py --model Devstral --ctx 131072 -y -- --verbose

Adding profiles for new models

Drop a new YAML file into settings/. The filename doesn't matter; the patterns: list does. The longest pattern that appears as a substring of the model filename wins.

# settings/my-model.yaml
display_name: "My Model"
patterns:
  - my-model
  - my-model-base

max_context: 131072
recommended_kv_quant: q8_0

sampling:
  temperature: 0.7
  top_k: 40
  top_p: 0.9
  min_p: 0.05
  repeat_penalty: 1.05

# Optional:
chat_template: chatml
extra_args:
  - --no-context-shift
notes: >
  Anything you want to remind yourself about this model.

Profiles with empty patterns: become the fallback when nothing else matches. See settings/_default.yaml.

Profiles currently bundled (arch string read from GGUF metadata):

Profile file	Models	llama.cpp arch
gemma-4.yaml	Gemma 4 E2B/E4B/12B/26B-A4B/31B	gemma4
qwen3_8.yaml	Qwen3.8-27B VLM + Qwen3.8-2.4T-A95B text MoE	qwen35 / qwen35moe
mellum.yaml	JetBrains Mellum2-12B-A2.5B (Base/Instruct/Thinking), MoE	mellum
exaone-4_5.yaml	LG EXAONE 4.5 33B VLM (dense, non-commercial license)	exaone4
step35.yaml	StepFun Step 3.5 Flash + Step 3.7-Flash (MoE ~196–198B/11B, MTP-3)	step35
granite-embedding-r2.yaml	IBM Granite Embedding Multilingual R2 97m/311m (embedding, not chat)	modern-bert
muse-glimmer.yaml	Meta Muse Glimmer 30B + optional vision/DFlash	muse-glimmer
minimax-m3.yaml	MiniMax-M3 428B-A23B multimodal MSA MoE	minimax-m3
glm-5.yaml	GLM-5/5.1	glm5
glm-5_2.yaml	GLM-5.2, 1M + IndexShare/MTP	glm-dsa
granite-switch-4_1.yaml	IBM Granite Switch 4.1 adapters	graniteswitch
deepseek-v4.yaml	DeepSeek-V4 Pro / Flash, 1M context	deepseek4
shieldstral.yaml	Shieldstral 1.0 3B safety classifier	Ministral 3-derived
ling-3.yaml	Ling 3.0 Flash (profile ready; compatible fork required)	not in mainline b10362
kimi-k3.yaml	Kimi-K3 (profile ready; compatible fork required)	proposed kimi-k3

Notes on the new profiles:

• Qwen3.8 keeps a separate profile from Qwen3.5/3.6 because its official coding evaluations use the thinking defaults (temp 1.0, top_p 0.95, top_k 20) instead of Qwen3.6's older coding temperature. The 27B variant is multimodal and can disable thinking; the 2.4T-A95B variant is text-only and always thinks. Both retain the existing qwen35/qwen35moe GGUF architecture family, so filename patterns deliberately take precedence without claiming those ambiguous architecture fallbacks.
• Mellum 2 is a code-focused MoE (64 experts, 8 active; 12B total / 2.5B active; 128k ctx). ngram_method is deliberately set to ngram-map-k4v (MTP-compatible) so it survives whether or not llama.cpp's mellum loader executes JetBrains' MTP head.
• EXAONE 4.5 is a dense 33B VLM. Its integrated MTP/NextN tail blocks are loaded but not executed by llama.cpp (b9500, same as exaone-moe), so the profile uses draftless ngram-mod, not draft-mtp. License is non-commercial (research/academic only).
• Step 3.5 / 3.7-Flash both load under step35 and both carry a real MTP-3 head (num_nextn_predict_layers), so the profile pairs draft-mtp + ngram-map-k4v with draft_max: 3. ⚠️ At ~196–198B MoE these exceed this machine's 48 GB (VRAM+RAM) — realistically need heavy --n-cpu-moe offload or are not runnable; the profile is for correctness/future smaller builds.
• Granite Embedding R2 is an embedding model — it runs as an embedding endpoint (--embeddings --pooling cls, set via extra_args), not a chat/completion model. Sampling/draft fields are inert in that mode.

How the auto-tuning works

1. Detect: total / free RAM, every GPU's total / free VRAM, total CPU cores.
2. Place the model: full GPU offload if it fits, else partial offload using the GGUF's exact n_layers, else CPU only.
3. Compute the KV budget: free VRAM (after the model) plus free RAM (minus a safety reserve).
4. Pick KV quant + context: try q8 → q5 → q4, pick the highest quality that fits the profile's max_context. Round context down to a multiple of 1024.
5. Threads / batch: scale with placement (full GPU offload needs fewer CPU threads than CPU-only inference; long context wants smaller batches to keep prompt-prefill memory bounded).
6. Multi-GPU: a model that fits the largest card alone is pinned to it (other GPUs hidden via the visibility env vars, so they stay free for gaming/OBS); larger models spread across all GPUs with a priority-weighted --tensor-split (priority × free VRAM), and the highest-scoring card becomes --main-gpu.
7. Hand authority to the AutoTuner: --fit off is always emitted so llama.cpp's own auto-fit pass never silently re-tunes the values the AutoTuner computed and logged. An overcommit fails loudly (OOM) instead of being quietly downscaled.

Project layout

auto_tuner/
├── auto_tuner.py        # main entry: terminal menu + glue
├── qt_launcher.py       # Qt GUI (model picker + sticky options + OCR dialog)
├── ocr_workflow.py      # shared GUI/TUI PDF/Office/image OCR pipeline
├── model_benchmark.py   # bounded real-server performance search + scoring
├── hardware.py          # CPU + multi-vendor GPU detection
├── scanner.py           # GGUF scanner: mmproj/draft pairing, capability detection
├── settings_loader.py   # YAML profile loader and matcher
├── tuner.py             # config calculation + llama-server command builder
├── launcher.py          # subprocess + Ctrl+C handling (Windows + Unix)
├── app_settings.py      # persistent GUI prefs (autotuner_settings.json)
├── startup_manager.py   # Windows/Linux/macOS login autostart integration
...
├── settings/
│   ├── _default.yaml
│   ...
│   ├── ministral.yaml
│   ├── bonsai.yaml
│   ...
├── requirements.txt
└── README.md

Building llama.cpp and forks

AutoTuner ships launch logic, not a bundled llama.cpp binary. Put your llama-server / fork build in one of the configured build folders (GUI: llama Builds, or LLAMA_CPP_DIR) and AutoTuner discovers it. The repo keeps the build recipes in separate scripts so this README stays short:

File	Purpose
llama_build.txt	Mainline/nightly llama.cpp Vulkan build (Windows/AMD-friendly; exact nightly tags use bNNNN_llama.cpp, untagged master uses bNNNN_dev_COMMIT_llama.cpp; runtime also verifies the current X.Y.Z-dev line).
llama_prerelease_build.txt	Official semantic release build (historical filename; resolves the newest published stable vX.Y.Z automatically, currently v0.2.0, and verifies its corresponding nightly bNNNN identity).
turboquant_llama_build.txt	TurboQuant KV-cache fork (tq_bXXXX_llama.cpp).
ternary_bonsai_llama_build.txt	PrismML Ternary/Bonsai fork (2b_bXXXX_llama.cpp), including the old-fork OpenSSL workaround.
diffusion_llama_build.txt	DiffusionGemma PR build with Vulkan.
diffusion_hip_llama_build.txt	DiffusionGemma HIP/ROCm build for AMD when Vulkan hits the ~1 GiB single-allocation limit.
setup_llamacpp_cuda.ps1	Windows NVIDIA/CUDA one-shot setup helper (PowerShell/Admin; freak288-style script).
setup_llamacpp_turboquant_cuda.ps1	Windows NVIDIA/CUDA TurboQuant setup helper.

The *.txt build recipes are PowerShell commands for the documented local workspace (L:/LAB/ai-local on the documented Pandaking setup). They handle both llama.cpp UI layouts automatically (tools/ui since b9174, tools/server/webui on older forks) and fall back to the prebuilt UI if the fork does not ship UI sources. Both mainline recipes deliberately keep full Git history (so the embedded numeric compatibility build is not incorrectly reported as b1). The normal master/nightly recipe now distinguishes an exact b-tag from an unreleased HEAD, reads the semantic base from upstream CMake, and checks the compiled binary's X.Y.Z-dev plus build number against git rev-list --count HEAD; it never labels a development build with the previous release tag. The semantic release recipe resolves the newest published stable vX.Y.Z, verifies the matching nightly bNNNN commit, and sets LLAMA_BUILD_IS_DEV=OFF, making llama-server --version report the exact semantic version plus its real build number for AutoTuner's feature gates. The CUDA setup helper likewise creates a fresh truthful folder instead of pulling newer source into an older bNNNN folder.

Ubuntu/Linux users can either build upstream llama.cpp normally or adapt the same CMake flags from the recipes. The only AutoTuner requirement is that the resulting binary is discoverable, e.g. LLAMA_CPP_DIR=/opt/ai-local/b9888_llama.cpp with build/bin/llama-server inside.

Server features (compatible through v0.2.0 / b10572)

Build/version probing and real server launches are validated against official v0.2.0 (b10566) and main/nightly 0.2.0-dev (b10572). The following llama-server features are supported (verified against llama-server --help / tools/server/README.md; the detailed historical source audit remains below):

Flag	Support
-fa [on|off|auto]	✅ Emits -fa on or -fa off explicitly; model profiles such as Unlimited-OCR can require the reference non-FA path
-ctk/-ctv f16/q8_0/q4_0/q4_1/q5_0/q5_1/iq4_nl	✅ All in the dropdown
--fit off	✅ Always emitted so llama.cpp's own auto-fit pass (default on) doesn't silently re-adjust the computed values (AutoTuner is the authority)
--perf	✅ Explicitly asserts performance timings so fork defaults cannot hide prompt/eval tokens/s; users can append --no-perf
--metrics	✅ Prometheus endpoint GET /metrics, including b10282+ speculative draft/accept counters (see "Monitoring")
--slots / --no-slots	✅ Emitted explicitly so the Expert toggle remains authoritative even though current mainline defaults /slots on
--cache-ram / -cram	✅ Host-memory prompt caching (PR #16391), auto-on with a 2048 MiB bounded default, adjustable in GUI/CLI and included in RAM planning. Vision caching is enabled for b10045+ and forced to 0 for older/unprobeable builds.
--reasoning on/off/auto	✅ Via dropdown
--reasoning-budget N	✅ Via spin-box. Renamed from --think-budget at b9625 (the old spelling is gone, not an alias); AutoTuner emits the new name and still reads the legacy one back from older persisted settings
--reasoning-preserve	✅ Optional Expert checkbox; omitted means template default
--chat-template-kwargs ...	✅ The dropdown produces this automatically
--jinja	✅ Ticked visibly
-lm, --load-mode {none,mmap,mlock,mmap+mlock,dio}	✅ Complete Expert dropdown. b10151's non-mmap mlock and explicit mmap+mlock semantics are version-gated; legacy checkbox snapshots are migrated.
-md external drafter	✅ Without --spec-type — the presence of -md enables the draft path automatically in mainline (verified b9442)
--spec-type draft-mtp	✅ Integrated/sidecar MTP (Qwen3-Next, Qwen3.6-MTP, GLM-4.7/5.2, DeepSeek V3.2 etc.)
--spec-type draft-eagle3 / draft-dflash / draft-dspark	✅ Architecture/tensor-aware sidecar detection. DSpark is emitted on b10164+; older binaries lose the entire DSpark path rather than mis-running it as DFlash
--spec-type ngram-mod (draftless)	✅ Via ngram_method: ngram-mod (default). Suppressed on MTP models because draft-mtp,ngram-mod crashes mid-generation (#23154, still open as of b9442)
--spec-type ngram-map-k4v (draftless)	✅ The MTP-compatible ngram method from ggerganov's MTP cleanup (PR #23269). Via ngram_method: ngram-map-k4v it runs together with draft-mtp → this is how you combine "MTP + ngram"
--spec-type ngram-map-k / ngram-simple / ngram-cache	✅ Selectable via ngram_method; only the type token is emitted, sub-parameters are left to the llama.cpp defaults
--spec-draft-n-max	✅ Via draft_max in the YAML profile; overridable per model in the Expert panel ("Speculative decoding" → draft n-max, 0 = profile default) since v4.9.6
--spec-draft-p-min	✅ Via draft_p_min in the YAML profile — the mainline default has been 0.0 since PR #23269; AutoTuner still emits an explicit 0.75 in both spec paths (external + integrated) so MTP only fires on confident steps
--spec-ngram-map-k4v-size-n/-size-m/-min-hits	✅ Via ngram_k4v_size_n / ngram_k4v_size_m / ngram_k4v_min_hits in the YAML (defaults 16/24/1 from PR #23269)
--spec-draft-ngl	✅ Always 99 (keep the MTP head on GPU)
--n-cpu-moe / --override-tensor	✅ --n-cpu-moe active; -ot prepared for targeted expert placement
--tensor-split / --main-gpu	✅ Priority-weighted for dense, capacity-fill for MoE, with single-GPU pinning; for multi-server the 2nd/3rd model is pinned to the emptier card via HIP_/GGML_VK_VISIBLE_DEVICES. A manual hard-pin to one card is available three ways — CLI --gpu NAME, the toolbar GPU dropdown, or the forced_gpu key in the settings JSON — all resolving through compute_config(force_gpu=…)
-mmdev, --mmproj-device DEVICE	✅ b10541+: automatically pins the complete multimodal projector to AutoTuner's selected post-visibility-remap main_gpu, matching the VRAM budget. Safely pruned for older/forked binaries that do not advertise the flag.
--rope-scaling yarn	✅ Already present
--numa	✅ Already present
--no-context-shift	✅ No longer duplicated (dedup via a seen-set)
--tools-runtime docker:…	✅ Correct value parsing/capability pruning through Extra CLI flags; never auto-enabled because it executes tools across a Docker/host trust boundary
Unlimited-OCR / DeepSeek-OCR MTMD	✅ Separate prompt/profile handling despite their shared deepseek2-ocr architecture; b10287+ Unlimited gate and stale-projector warning; shared GUI/TUI image/PDF/Office workflow; F16 KV, -fa off, DRY guard, and normal /v1/chat/completions API

Review b10441 → b10549

Reviewed all 108 upstream commits from exact tag b10441 (0177dcc7) through b10549 (b2e5e9b2). Full source/build evidence is in docs/llama-b10549-audit.md.

• Integrated: b10541 --mmproj-device keeps MTMD on the same exact GPU that owns AutoTuner's projector VRAM budget, including visibility-remapped dual-GPU launches; old binaries lose the complete option/value pair through help-based pruning.
• Current model support: Kimi-K3's b10448 mainline text loader is now build-gated without falsely claiming K3 vision; Ling 3.0 Flash/Tiny use the native b10460 bailingmoe3 loader, official sampling, hybrid KV count, and integrated/sidecar MTP detection.
• Automatic rebuild benefits: DSpark speculator formats/LFM2 targets, b10549 LFM2/LFM2MoE tensor split, Granite SWA/MoE SWA metadata, repeated MTMD prompt caching, DeepSeek-OCR, Vulkan Q8-KV/FA, HIP, CUDA, Metal, SYCL, OpenCL, and server/router fixes need no additional launch controls.
• Build-number mismatch fixed: master can legitimately be commits ahead of the newest release (build 10548 while b10545 is still latest). The recipe now uses the embedded full-history commit count, accepts only an exact HEAD tag, marks untagged builds _dev_<commit>, and verifies the compiled --version; no hard-coded offset or waiting is required.
• Not promoted: router preset-only dedup-cache-models remains a preset concern, not a normal AutoTuner performance control.

Review b10329 → b10441

Reviewed all 112 upstream commits from exact tag b10329 (18f7ad7f) through b10441 (0177dcc7). The source, official-package, backend, architecture, and memory-accounting evidence is documented in docs/llama-b10441-audit.md.

• No emitted flag broke: the complete AutoTuner server surface remains in b10441. --load-mode auto is now the upstream default; the new value-bearing --reasoning-effort is safe through Extra CLI flags.
• Backend identity is now exact: CUDA, HIP, Vulkan, SYCL, Metal, and OPENVINO device prefixes from the selected binary are retained. CUDA/SYCL no longer inherit Vulkan indices or selectors.
• Unified memory is one pool: Apple Silicon and confirmed integrated GPUs use live available memory, with CPU/GPU allocations counted once. Full-GPU KV no longer receives an impossible host-RAM supplement.
• Architecture updates: existing Muse Glimmer and Granite Switch profiles cover their new native loaders; MiniMax-Text-01/MiniMax-M1 gains a minimax-01 hybrid-MoE profile; PocketTTS remains a dedicated TTS workflow, not a normal text-chat claim.
• Official packages expanded: b10441 publishes Windows x64 Vulkan, ROCm, SYCL, OpenVINO, CUDA 12.4 and CUDA 13.3 builds, plus native macOS arm64 and Ubuntu backend packages. AutoTuner launches these external binaries rather than linking to one backend.

Review b10151 → b10329

Reviewed all 178 upstream commits from exact tag b10151 (8e8681e0) through b10329 (18f7ad7f). The complete evidence and scope decisions are documented in docs/llama-b10329-audit.md.

• Integrated: DSpark sidecar/tensor detection and draft-dspark build gating; Unlimited-OCR's b10285 multi-row batching + b10287 32-tile fix; deterministic OCR profile; complete GUI/TUI PDF/Office/image workflow; b10329 value-bearing flag handling including --tools-runtime; explicit Flash Attention off; F16 KV profile opt-in; selected-server alias/process verification before any document upload.
• Automatic benefits: new speculative counters appear in the already-enabled /metrics; Qwen3-Next/DeepSeek V3.2/GLM MTP loaders, EAGLE-3 v3, model/router, Vulkan/ROCm/Metal/CUDA/SYCL, tokenizer, and MTMD fixes require only a rebuilt llama.cpp binary.
• Not automatic: Docker tool isolation and MCP configs remain trusted Extra CLI choices. The separate llama-tts breaking changes do not affect AutoTuner's server runners. b10329 only announces a future upstream port change; AutoTuner continues to pass explicit port 1234.

Review b10107 → b10151

Reviewed all 44 upstream commits from tag b10107 (c0bc859) through b10151 (8e8681e). The only new/expanded launcher inputs are the split --load-mode semantics and two experimental stdio-MCP configuration flags.

• Integrated: Expert model-load dropdown for none, mmap, non-mmap mlock, mmap+mlock, and dio; compatibility adaptation for pre-b10151 binaries; current-build GPU locking no longer hits the old unconditional veto.
• Not promoted to tuning controls: --mcp-servers-config PATH and --mcp-servers-json JSON spawn external processes and change CORS behavior. They are server integrations rather than performance knobs, remain available through Extra CLI flags, and should only be used with trusted configs.
• Automatic upstream benefits: explicit -md now wins over discovered draft sidecars; reasoning budgets recognize multiple end sequences; MiniMax-M3/GLM indexer and backend/KV fixes require no new AutoTuner setting.
• No change required: KV types/cache sizing, context/batching, GPU-layer offload, tensor split, and speculative token-count flags did not gain new controls or incompatible defaults in this range.

Review b9963 → b10056

Reviewed all 93 upstream commits through tag b10056 (b85833e). No AutoTuner-emitted server flag was removed, renamed, or changed incompatibly. Changes integrated in this review:

• Vision prompt caching: current mtmd state handling can reuse repeated image prompts. AutoTuner enables --cache-ram for b10045+ and keeps older or unprobeable builds on the safe --cache-ram 0 path. A real b10058 Gemma 4 + mmproj test returned cached_tokens 0 → 279 and reduced the repeated request from 3.13 s to 0.30 s.
• /slots toggle fixed: b10056 defaults /slots on, so AutoTuner now emits --slots or --no-slots explicitly instead of treating omission as off.
• Reasoning history: --reasoning-preserve is available in the Expert panel and persists with the other per-model Expert settings.
• Hy3/Hy-MT2: hy_v3 + MTP support merged in b9993 (PR #25395), so the profile no longer tells users to select a PR fork.
• New optional upstream surface: --cors-origins, --cors-methods, --cors-headers, and --cors-credentials landed in b10010. They remain available through Extra CLI flags; a dedicated four-field UI is unnecessary while AutoTuner binds 127.0.0.1 by default.

Transparent rebuild benefits include Minimax2 EAGLE-3 support, Vulkan native MXFP4/NVFP4 conversions, prompt-cache/checkpoint fixes, DeepSeek V4 graph optimisations, mtmd fixes, and CUDA/HIP/SYCL backend improvements.

Review b9840 → b9888

Reviewed mainline up to b9888 (cb295bf, CUDA FlashAttention K/V cache-type validation). No AutoTuner flag was removed or renamed upstream. Changes made for v4.7.9:

• Terminal throughput visibility asserted: AutoTuner emits --perf for normal llama-server, llama-diffusion-cli, and llama-diffusion-gemma-server, so fork defaults cannot hide prompt/eval timings and tokens/s. Current mainline defaults these timings on; --metrics remains enabled for machine-readable monitoring.
• NVIDIA CUDA safety: b9888 validates V-cache types for CUDA FlashAttention too. Since default CUDA builds have GGML_CUDA_FA_ALL_QUANTS=OFF, AutoTuner keeps automatic KV choices symmetric on NVIDIA (high- and low-VRAM) while preserving AMD/Vulkan asymmetric K/V choices for extra context. Expert-mode manual K/V pins still pass through unchanged.
• Tracked settings removed: autotuner_settings.json is now only local user state (already gitignored) and is removed from Git tracking for GitHub releases.

Relevant upstream commits in this range are backend/runtime fixes (CUDA Gemma E4B MTP FA, stale tensor-split params for draft models, tensor-parallel + --n-cpu-moe, Vulkan integer overflow, UI/MCP fixes). They do not require new AutoTuner flags beyond the --perf verbosity fix above.

Review b9625 → b9840

Reviewed the range up to b9840. Every server flag and --spec-type value in use was verified against the current tools/server/README.md (--help table) and docs/speculative.md. No existing flag was removed or renamed — the AutoTuner's flag surface is unchanged and still valid. Changes this round are AutoTuner-side additions and fork-build fixes:

• EAGLE-3 speculative decoding (PR #18039; Qwen3.5/3.6 since PR #24593 / b9723). --spec-type draft-eagle3 is now emitted automatically when the paired drafter GGUF declares general.architecture = eagle3 (a one-layer transformer that reads the target's hidden states — higher acceptance than a plain draft of the same size). Sibling files named *-eagle3* are auto-paired like any draft; scanner.py reclassifies an eagle3-arch GGUF into the draft pool (never listed as a choosable model).
• DFlash speculative decoding (PR #22105). --spec-type draft-dflash is emitted automatically when the paired drafter declares general.architecture = dflash (block-diffusion; emits a whole block per step). If auto-pairing misses a custom filename, pick the DFlash GGUF in the GUI's draft dropdown; it is labelled [DFlash] and remembered per model.
• Fork discovery hardened. Versioned fork dirs (2b_b8840_llama.cpp, tq_b9632_llama.cpp, …) now resolve correctly — a profile hint like 2b_llama/llama-server matches the on-disk 2b_b8840_llama.cpp after normalizing the _b<NUM> version segment. The 1-bit (1b_) and 2-bit/Ternary (2b_) Bonsai families stay distinct. Forks that match the name pattern but have no built llama-server binary are now reported in the llama_cpp debug category (instead of vanishing silently), and the terminal launcher now finds L:/LAB/ai-local (the documented workspace) even without LLAMA_CPP_DIR set.
• bonsai-ternary.yaml corrected: server_binary now points to 2b_llama (2-bit/Ternary fork), not 1b_llama (1-bit Bonsai).
• Build scripts (*_build.txt) now probe BOTH UI layouts — pre-b9174 tools/server/webui/ and post-b9174 tools/ui/ — and fall back to the HF prebuilt UI when neither exists. The Bonsai (b8840-basis) build adds -DLLAMA_OPENSSL=OFF to work around the cpp-httplib 0.40.0 / OpenSSL 3.2+ C2440 const error.

Everything else AutoTuner emits is unchanged and still valid at b9840: --fit [on|off], -fa [on|off|auto], --cache-ram/-cram, --metrics, --n-cpu-moe/-ncmoe, --tensor-split, --main-gpu, YaRN, KV-cache types, --reasoning/-rea, --reasoning-budget, --chat-template-kwargs, --jinja, --mlock/--no-mmap, and the full speculative set.

Review b9500 → b9625

Reviewed the range up to b9625. Every server flag and --spec-type value in use was verified against the b9625 common/arg.cpp and common/speculative.cpp. One breaking change affected the AutoTuner and is fixed this round:

• --think-budget renamed to --reasoning-budget (CLI). At b9625 the reasoning token-budget flag is {"--reasoning-budget"} "N" (-1 unrestricted / 0 immediate end / N>0 budget); the old --think-budget spelling is gone — not kept as an alias (the env var stays LLAMA_ARG_THINK_BUDGET, and the short reasoning toggle gained a -rea alias). The Expert panel's spin-box now emits --reasoning-budget, and _parse_reasoning_from_extras reads both the new and the legacy name so older autotuner_settings.json files still restore the spin-box correctly. A sibling --reasoning-budget-message MESSAGE was also added (text injected before the end-of-thinking tag when the budget is exhausted) — not emitted by AutoTuner.

Everything else AutoTuner emits is unchanged and still valid at b9625, re-confirmed against the source: --fit [on|off], -fa [on|off|auto], --cache-ram/-cram (-1 no-limit / 0 disable), --metrics, --n-cpu-moe/-ncmoe, --tensor-split, --main-gpu, --rope-scaling yarn + --rope-scale, --numa, --mlock/--no-mmap, --no-context-shift, --parallel, --jinja, --reasoning/-rea, --chat-template-kwargs, --mmproj, the sampler flags, and the full speculative set — --spec-type with the tokens draft-mtp, ngram-mod, ngram-map-k, ngram-map-k4v, ngram-simple, ngram-cache, plus --spec-draft-ngl/-n-max/-p-min, --spec-ngram-mod-n-match/-n-min/-n-max, and --spec-ngram-map-k4v-size-n/-size-m/-min-hits. The -md external drafter still enables the draft path without an explicit --spec-type.

Full review of all 58 commits between b9442 and b9500. Result: no functional AutoTuner changes to existing flags needed — every server flag and --spec-type value in use was verified against the b9500 common/arg.cpp and common/speculative.cpp and is unchanged and still valid. Specifically re-confirmed present at b9500: --spec-type, --spec-draft-ngl/-n-max/-p-min, --spec-ngram-mod-n-match/-n-min/-n-max, --spec-ngram-map-k4v-size-n/-size-m/-min-hits, and the spec-type tokens draft-mtp, ngram-mod, ngram-map-k, ngram-map-k4v, ngram-simple, ngram-cache. Relevant points:

• Speculative: draft-simple auto-enable removed (#23988). The server no longer auto-enables a draft-simple path; the common/arg.cpp diff was whitespace-only (no flag renamed/removed). No impact — AutoTuner always emits --spec-type explicitly and never relied on auto-enabling. A new draft-eagle3 spec-type also exists now (EAGLE3 drafters); AutoTuner now emits it when an eagle3-arch drafter is paired (see b9625→b9840 review below). AutoTuner.
• Gemma 4 "unified" runtime + 12B (#24077, #24082, #24088, #24025). Vision/audio (mtmd) fixes for the encoder-free "unified" Gemma 4 and the new arch enums gemma4uv/gemma4ua. The 12B gemma-4-12b-it is the unified variant, but its language-model GGUF still loads under general.architecture = gemma4 (verified in the b9500 converter: Gemma4UnifiedModel → MODEL_ARCH.GEMMA4); gemma4uv/gemma4ua are only projector types on the separate mmproj file. → scanner, KV-sizing and match_profile treat the 12B exactly like the rest of the family. The gemma-4.yaml profile was updated for accuracy (12B added to the context-tier comment and the multimodal/audio notes; throughput-vs-dense caveat clarified) — no code change required for it to work.
• Qwen3.5 MTP post-norm (#24025). Qwen35 now uses the post-norm hidden state for MTP, internal rename pre_norm → nextn. Runtime correction only; no CLI flag or metadata-key change → the tri-state MTP scanner over {arch}.nextn_predict_layers stays valid.
• New architectures (profiles added this round). mellum (JetBrains Mellum2-12B-A2.5B, MoE, #23966), exaone4 (EXAONE 4.5 33B VLM, #21733), step35 (StepFun Step 3.5 + Step 3.7-Flash, MoE+MTP-3, #23274/#23845), and modern-bert (IBM Granite Embedding Multilingual R2 97m/311m, #22716). See Adding profiles for new models — these are profile additions, not forced by any flag rename. The arch is read dynamically from the GGUF metadata.
• Vulkan performance (transparent). Device mutex no longer held while compiling pipelines (#23641), reduced host-memory lock contention (#23376), Q3_K/Q6_K block-load on 32-bit ints (#23056). Benefits from the rebuild alone, no flag change; relevant to the Vulkan backend on the R9700 / RX 9070 XT (faster server start / pipeline warmup).
• Library API (not CLI). llama_set_warmup deprecated (#24009), llama_context max-outputs limited (#23861), CUDA reserves quantized-KV space at startup (#23907). No effect on the flags AutoTuner emits.

Scanner fix shipped this round (AutoTuner-side): scanner.py's _ROPE_SCALE_SUPPORTED_ARCHS matched only the qwen2 prefix, so the newer qwen3* / qwen35* arch strings (Qwen3/3.5/3.6) fell through and were excluded from automatic YaRN (they could only get RoPE-scaling via an explicit rope_scale.enabled: true in the profile). Broadened the prefix qwen2 → qwen (matched via startswith, so it now covers qwen/qwen2*/qwen3*/qwen35* and stays correct for future Qwen archs).

Review b9409 → b9442

Reviewed the releases up to b9442 (d4c8e2c, a vocab/tokenizer commit adding jina-embeddings-v2-base-zh). Result: no functional AutoTuner changes to existing flags needed — every server flag in use (--fit off, --metrics, --cache-ram, --spec-type with draft-mtp / ngram-mod / ngram-map-k4v, --spec-draft-*, -fa on, --n-cpu-moe, --tensor-split, YaRN, KV-cache types) was verified against the b9442 common/arg.cpp and common/speculative.cpp and is unchanged and still valid. The changes added in this round are AutoTuner-side, not forced by any flag rename:

• mmproj detection now matches the mmproj marker anywhere in a filename and also picks up .mmproj-extension projectors, so the MXFP4 MoE pair (…-mxfp4-moe-mmproj-f16.gguf) is paired correctly.
• GGUF general.sampling.* is now read and used to fill any sampler value a matched profile leaves unspecified — fixing repetition loops and broken tool-calls on models without a tailored profile.
• MoE multi-GPU spread switched from priority-weighting to capacity-fill, so both GPUs are packed with expert layers instead of stranding VRAM on the secondary card.

Review b9371 → b9409

Reviewed the releases up to b9409 (fe12e42, a pure sync : ggml commit). Result: no functional AutoTuner changes to existing flags needed — all server flags in use (--fit off, --metrics, --cache-ram, --spec-type, --spec-draft-*, -fa on, --n-cpu-moe, YaRN, KV-cache types) were unchanged and still valid. Added this round (not forced by a b9409 flag rename, but as a feature):

• --cache-ram prompt caching is now actively emitted (previously not at all). At the time this review was written it was conservatively limited to non-vision models; the b10056 review above adds build-gated Vision support.
• Multi-server port assignment (1234, 1235, … with a reset on exit) and live-VRAM load-balancing onto the emptier GPU before starting a second/third model — purely GUI/launcher-side, no new server flags.

Review b9334 → b9371 (37 commits)

Full review of all 37 commits between b9334 and b9371. Result: no functional AutoTuner changes needed — all server flags in use (--fit off, --metrics, --spec-type, --spec-draft-*, -fa on, --n-cpu-moe, YaRN, KV-cache types) were unchanged. Relevant points:

• Env rename (#23778): llama.cpp moved several environment variables to the unified LLAMA_ARG_ prefix: LLAMA_LOG_FILE/COLORS/VERBOSITY/PREFIX/TIMESTAMPS → LLAMA_ARG_LOG_*, LLAMA_OFFLINE → LLAMA_ARG_OFFLINE, LLAMA_CHAT_TEMPLATE_KWARGS → LLAMA_ARG_CHAT_TEMPLATE_KWARGS. The CLI flags themselves stay the same. AutoTuner sets only HIP_VISIBLE_DEVICES / GGML_VK_VISIBLE_DEVICES as env overrides (GGML vars, not affected); LLAMA_ARG_FIT already carries the prefix → no impact. ⚠️ If llama log/offline env overrides are added in the future, use the LLAMA_ARG_ prefix from b9371 on.
• Vulkan performance: several transparent backend optimisations (MUL_MAT_VEC 4 K/iteration for F16/F32 #22887, conv2d + coopmat1 #22620, REPEAT f16→f16 #23298). Benefits from the rebuild alone, no flag change. The AMD UMA transfer-queue fix (#22455) affects only integrated GPUs/APUs, not the dedicated R9700 / RX 9070 XT.
• New model/conversion support (convert-side, not server runtime): Gemma4ForCausalLM conversion (#23682), MiniCPM5 tokenizer (#23384), talkie-1930-13b (#22596), Mistral3-NVFP4 weight scales (#23629). Profile maintenance only on adoption — the arch is read dynamically from the metadata in the tuner.
• Server code: cosmetic only (SSL log message #23393, cpp-httplib 0.46.0 #23650).

Speculative decoding (MTP + n-gram)

The AutoTuner combines up to three speculative paths into one --spec-type list:

• Path A — external sibling drafter (-md): a small *-draft-* / *-assistant-* sibling model. Skipped when vision (--mmproj) is active (three large graphs in VRAM at once is too risky on 16-GB cards).
• Path B — integrated MTP (--spec-type draft-mtp): the trained MTP head lives inside the main GGUF (Qwen3.6-MTP etc.). Coexists with vision since b9180.
• Path C — draftless n-gram (--spec-type <ngram_method>): needs no draft model. Method selectable per profile via ngram_method.

Since b9334 the draftless family has grown: ngram-mod (default), ngram-map-k, ngram-map-k4v, ngram-simple, ngram-cache.

MTP + n-gram together. Only ngram-mod conflicts with draft-mtp (draft-mtp,ngram-mod → random mid-generation crashes, llama.cpp #23154, still open as of b9442). That's why the tuner suppresses ngram-mod next to MTP. The ngram-map-* methods were built by ggerganov's MTP cleanup (PR #23269) specifically to coexist with draft-mtp — ngram-map-k4v is even in its --spec-default. So to enable "MTP + n-gram" on an MTP model, one profile entry is enough:

# settings/qwen3_5-3_6.yaml  (Qwen3.6-MTP)
ngram_method: ngram-map-k4v     # runs next to draft-mtp instead of being suppressed
# optionally fine-tune (defaults from PR #23269):
ngram_k4v_size_n: 16
ngram_k4v_size_m: 24
ngram_k4v_min_hits: 1

For an MTP model with ngram_method: ngram-map-k4v this yields:

--spec-type draft-mtp,ngram-map-k4v
--spec-draft-n-max 2 --spec-draft-ngl 99 --spec-draft-p-min 0.75
--spec-ngram-map-k4v-size-n 16 --spec-ngram-map-k4v-size-m 24 --spec-ngram-map-k4v-min-hits 1

An unknown ngram_method value in the YAML falls back to ngram-mod with a warning at load time (instead of crashing only at server start).

⚠️ Reality check: on bandwidth-limited MoE-A3B models, speculative decoding can fail to beat baseline (expert saturation). n-gram is now default-on, but remains user-toggleable; use the Performance test's separate prompt-processing and n_decode results to decide whether to disable it.

Several models at once + GPU load-balancing

The GUI can run multiple llama-server instances in parallel:

• Automatic port assignment. The base port entry (default 1234) applies to the first server. Each additional server gets the next free port: 0 running → 1234, 1 → 1235, 2 → 1236, … Ports are checked before assignment (socket bind) so there's no collision with other processes.
• Counter reset on exit. When a server is stopped or crashes, the launcher frees its port again — the next launch reuses the freed port. The counter is always base + number of running servers.
• Load-balancing before the 2nd/3rd model. Before an additional model starts, the AutoTuner re-reads the current VRAM usage (i.e. including what already-loaded models hold) and steers the new model onto the emptier card — exactly that GPU is made visible via HIP_VISIBLE_DEVICES / GGML_VK_VISIBLE_DEVICES.
• Clear refusal instead of overcommit. If the model no longer fits on any card (e.g. the R9700 already at 31/32 GB), the launcher aborts with a clear message and shows the VRAM usage of all cards, instead of overloading an already-full device. The first model still uses the normal automatic multi-GPU split (--tensor-split); the per-card check is only a warning, not a hard refusal, when a single model is meant to be split across both cards.

Host-RAM prompt caching (--cache-ram)

Since PR #16391 llama-server caches computed prompt prefixes in regular system RAM and swaps them back into the llama_context when a new request shares a long prefix (system prompt, RAG scaffold, Roo-Code preamble). This massively lowers time-to-first-token on repeated prompts.

• Auto-on with a bounded default cap of 2048 MiB. The GUI spinbox and CLI --cache-ram-mib accept a positive limit, -1 (unlimited), or 0 (disabled); the planned amount is subtracted from the RAM/KV budget.
• Switchable off via the Prompt caching checkbox in the Launch options or with CLI --no-prompt-cache (emits --cache-ram 0).
• Vision support on b10045+. Current builds deep-copy multimodal prompt state and reuse repeated image prompts. Older or unprobeable builds retain the conservative --cache-ram 0 fallback. The per-model choice is remembered like vision/draft/thinking; the MiB limit is global.

Choosing among several mmproj precisions

If a model ships projectors side by side (…-bf16, …-f16, …-f32), the scanner keeps all same-folder candidates. The always-visible mmproj dropdown lists each projector with its file size (just like external drafters), marks incompatible experiments, and remembers the choice per model in autotuner_settings.json (mmproj_selection). The automatic pre-pick prefers the highest precision (f32 > f16 > bf16) instead of taking bf16 purely alphabetically.

Intel CPU, SYCL, and OpenVINO

AutoTuner selects capabilities from the exact llama-server binary, not from an OS-vendor guess. This matters on Intel laptops and mixed workstations:

1. An x64 CPU build always remains the safe baseline. On a ThinkPad, thread count and RAM/KV sizing are derived normally even when the iGPU is too old or too small.
2. The b10441 SYCL backend supports Intel GPU families from 11th-generation Core onward, including Iris Xe, built-in Arc, and discrete Arc. Official b10441 Windows x64 and Ubuntu FP16/FP32 packages exist. AutoTuner retains SYCL0, uses ONEAPI_DEVICE_SELECTOR, and counts confirmed iGPU memory as shared system RAM. A 10th-generation or older iGPU is not promised; use CPU or a separately validated OpenCL path there.
3. The b10441 OpenVINO backend is available officially on Windows and Ubuntu and can target Intel CPU, GPU, or NPU. It is a whole-graph backend, represented as one host/unified-memory accelerator. Set the upstream environment variable before starting AutoTuner when a target is required: GGML_OPENVINO_DEVICE=CPU, GPU, GPU.0, or NPU. Model, quantization, stateful-execution, and NPU coverage remain experimental upstream.
4. OpenVINO cannot be mixed with a Vulkan/ROCm pool in one process. On a workstation with strong discrete cards, use a separate server process for an Intel iGPU/NPU background model. When the selected binary is SYCL or OpenVINO, unrelated OS-visible AMD/NVIDIA devices are excluded from that process's tuning calculation.

Discrete Intel Arc cards are no longer blanket-filtered when AMD/NVIDIA peers exist; only confirmed integrated/shared-memory Intel devices are treated as auxiliary in a mixed generic GPU pool.

Apple Silicon / M5 unified memory

The macOS release is built natively on GitHub's Apple-Silicon arm64 runner and published as AutoTuner-macOS-arm64.zip. Metal is enabled by default in the official llama.cpp macOS arm64 package. M1 through M5-family names require no hard-coded chip table.

RAM and VRAM are not separate on Apple Silicon. AutoTuner uses live available memory, caps it by the selected Metal runtime report, subtracts model, KV, projector/draft, prompt-cache, and workspace allocations once, and displays one unified total. low_vram may move KV compute to the CPU, but it cannot create extra physical memory on a unified-memory Mac.

Monitoring (/health + /metrics + optional /slots)

The Expert settings include diagnostics toggles for --metrics and --slots. Metrics stay enabled by default. AutoTuner emits --slots when monitoring is requested and --no-slots otherwise, because current mainline defaults the endpoint on while older builds differ. All endpoints use the same host:port as the inference API (there is no separate metrics port):

• GET /health — 503 while loading, 200 when the model is ready. The Qt GUI polls this endpoint and switches the status from Loading model to Ready (see above).

• GET /slots — when --slots is enabled, the Qt GUI polls this endpoint and shows a compact busy/total slot summary in the server dropdown.

• GET /metrics — Prometheus text format. The most important metrics (single-model mode, prefix llamacpp:):

  Metric	Type	Meaning
  llamacpp:predicted_tokens_seconds	gauge	Generation throughput (tok/s)
  llamacpp:prompt_tokens_seconds	gauge	Prompt/prefill throughput (tok/s)
  llamacpp:kv_cache_usage_ratio	gauge	KV-cache fill level (1.0 = 100%)
  llamacpp:kv_cache_tokens	gauge	Tokens in the KV cache
  llamacpp:requests_processing	gauge	Active requests
  llamacpp:tokens_predicted_total	counter	Generated tokens, cumulative
  llamacpp:prompt_tokens_total	counter	Prompt tokens, cumulative

  Scraping without a Prometheus client (e.g. for the System Tricorder):

  import urllib.request
  def llama_metrics(base_url: str) -> dict[str, float]:
      out = {}
      with urllib.request.urlopen(f"{base_url}/metrics", timeout=0.5) as r:
          for line in r.read().decode().splitlines():
              if line and not line.startswith("#"):
                  name, _, val = line.partition(" ")
                  try: out[name] = float(val)
                  except ValueError: pass
      return out
  # llama_metrics("http://127.0.0.1:1234")["llamacpp:predicted_tokens_seconds"]

• get_metadata.py — drop it into the folder with your models (pip install gguf) to read and save the metadata of every model. For debugging!

License

MIT.