Design spec: benchmark-first CPU text refinement (llama.cpp tier conditional)

[tonythethompson]

Summary

• Revised per review: the spec's original gap claim is corrected — the shipped manifest bundles Qwen2.5-1.5B for text refinement (bundled-models.manifest.json), CPU is an allowed provider (StageRuntimeRequirements.cs), and the router selects Qwen by default when registered; CPU users already get local refinement, so the premise is now "measure, don't assume"
• Reframed as benchmark-first: Phase 0 measures the shipped 1.5B int8/CPU path on reference low-end hardware (speed, RSS) plus a human-reviewed semantic quality suite — the existing QwenRefinementOutputGuard is treated as a safety filter, not a semantic quality measure
• Phase 1 (llama.cpp/GGUF for a larger model) is conditional on Phase 0 showing a quality gap with speed headroom, with route selection owned by the runtime planner (StageRuntimeRequirements/RuntimePlanner), not a router bypass; per-artifact license verification for community GGUFs per manifest schema
• Narrowed to text refinement only; translation and ASR dropped as separate scopes

Verification

• Docs-only change; no build impact
• Evidence citations verified against live code: manifest Qwen2.5-1.5B entry, TextRefinement stage requirements (CPU provider, int8 variants, TRT-RTX exclusion for GenAI families), router default selection

[cursor]

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• queue - adds this PR to the back of the merge queue
• fast - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[amazon-q-developer]

This design specification is well-structured and ready for product sign-off. The proposal to add llama.cpp/GGUF text refinement support for CPU-only and low-VRAM users addresses a clear coverage gap with a sound technical approach.

Key strengths:

• Problem statement clearly identifies the underserved user segment (CPU-only/low-VRAM)
• Technical design follows existing architectural patterns (managed sidecar, hardware-tier routing, model manifest integration)
• Comprehensive quality gates (QwenRefinementOutputGuard parity checks, DubBench metrics)
• Risk mitigation strategies address lifecycle concerns, quality degradation, and hardware detection edge cases
• Execution plan provides clear phased implementation with measurable acceptance criteria

The specification is complete and no blocking issues were identified. The open questions appropriately defer packaging and threshold decisions to implementation review.

---

You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

[Copilot]

Copilot review overview

🟡 Changes recommended

The specification misstates current runtime behavior and needs planner-aligned routing, model-governance requirements, and a valid semantic quality gate.

Review effort: Balanced
Findings: 6

Open (6)

• Correct runtime baseline before proposing a new hardware tier · New
• Reframe model performance gap and require benchmark evidence · New
• Add semantic quality metrics beyond output safety checks · New
• Include model governance metadata and pinned source revision · New
• Require provider availability and successful runtime smoke tests · New
• Use runtime planner output instead of hardcoded hardware probes · New

What changed in this PR

Proposes a llama.cpp/GGUF text-refinement tier for CPU-only and low-VRAM users.

Changes:

• Defines sidecar architecture, routing, readiness, and manifest integration.
• Proposes quality and performance validation.
• Documents rollout risks and open distribution questions.

File	Description
docs/​specs/​design-llamacpp-cpu-text-tier.md	Adds the llama.cpp CPU text-tier design specification.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tonythethompson]

Docs-only spec, so no build concerns. I checked its claims against the code at this head. The main issue is the problem statement: the current text-refinement model is a 1.5B model that is already allowed to run on CPU, so the spec needs measured numbers before it can justify a second runtime. Details are inline, one per thread.

[tonythethompson]

The premise doesn't match the code. The shipped text-refinement model is tonythethompson/Qwen2.5-1.5B-Instruct (in bundled-models.manifest.json, engine family qwen-instruct), not a 7–9B class export. CPU is also already an allowed provider for RuntimeStage.TextRefinement: StageRuntimeRequirements.cs (around lines 177–191) uses DefaultOnnxStageAllowedProviders, which ends with ExecutionProviderKind.Cpu. The manifest's Olive block also lists cpu and int4. So a local CPU option exists today, and the claim that there is no local option is not accurate.

That changes what the spec has to prove. Before adding a second native runtime, please add measured ORT GenAI CPU numbers for the shipped 1.5B model (and an int4 CPU variant) on a target laptop: time to first token, tokens per second, and wall time per segment batch. Show that it's actually too slow. The low-VRAM partial-offload case gets weaker too, because a ~1.5B model fits in 4 GB without -ngl tricks.

[tonythethompson]

TRT-RTX isn't on the text-refinement path. StageRuntimeRequirements.cs deliberately strips TensorRT and TRT-RTX from qwen-instruct with WithoutTensorRtFamilies(...), because ORT GenAI on NvTensorRtRtx crashes the process with a native stack overflow. On NVIDIA hardware, Qwen refinement plans to CUDA or DirectML, not TRT-RTX. The same mistake shows up in the tier table (line 62, "ONNX/TRT-RTX default") and in the non-goals. Please fix the engine row and the tier table so reviewers are comparing llama.cpp against the routing that actually ships.

[tonythethompson]

There's no DnnlEngineCacheProbe in the repo (the closest are EngineCacheProbe and Dnnl/DnnlReadinessProbe). Here's a pointer to the hardware-profiling types that do exist:

Suggested change

	**Hardware tier detection** reuses existing capability probing (`Trackdub.Inference.Onnx/Runtime` EP probes, `DnnlEngineCacheProbe`-style pattern) to classify:
	**Hardware tier detection** reuses existing capability probing (`Trackdub.Inference.Onnx/Runtime` device enumerators and `WindowsVramMonitor`, `Runtime/Planning/MachineHardwareProfileProvider`, and the cached EP readiness probes in `Runtime/Planning/CachingReadinessProbes.cs`) to classify:

[tonythethompson]

Tier selection belongs in the runtime planner, not RoutedTextRefinementEngine. Today the router only matches by alias and engine family. The hardware and EP decision happens in IStageRuntimePlanner, which QwenTextRefinementEngine calls through StageRuntimePlanningRequestFactory.ApplyPreferredModelTierAsync, then PlanAsync, then EnsurePlanReady. If the router adds its own hardware-tier probe, there are two sources of truth. The readiness report and the plan would say qwen-instruct on CPU or DML while the router quietly runs llama.cpp, and LastExecutionSummary and the G5 readiness panel would disagree with what actually ran.

Please make llama.cpp visible to the planner for RuntimeStage.TextRefinement, for example as its own engine family or runtime in the manifest and requirements catalog. Then one plan drives selection, readiness, and the execution summary, and the router stays alias-only.

[tonythethompson]

The quality baseline names a variant that doesn't ship. The manifest's default qwen-instruct variant is the GenAI bundle produced by tools/olive/Export-QwenTrtRtxGenAi.ps1, and that script exports -p fp16. int4 is only a supported Olive precision, not the shipped default. Please name the exact baseline (manifest revision f56ee652… plus variant alias). Otherwise the parity check in step 3 compares against an artifact users aren't running.

[tonythethompson]

QwenRefinementOutputGuard parity can't show quality parity. The guard is a per-segment reject filter: explanation text, format, multi-segment leakage, length, edit distance, and name/number checks, each falling back to the original text. It also marks unchanged output as not accepted. Two models can match on acceptance rate while one makes worse edits, and a model that is lazier about editing can look "safer." Please keep the guard as a floor (no higher rejection rate than baseline), and add an actual quality signal for the adopt-Q4_K_M decision: for example, WER or CER against reference transcripts on the fixed suite, or a blind A/B on the accepted outputs.

[tonythethompson]

OpenVINO EP vs CPU EP smoke results (Intel CPU sandbox)

Environment: same constrained-CPU sandbox as the TTS benchmark — Intel Xeon 6985P-C @ 2.30 GHz, no GPU/NPU. Stack: openvino 2026.4.1, onnxruntime-openvino 1.24.1 (ORT with OpenVINO EP + CPU EP). Caveat as before: server-class constrained environment, not a low-end laptop; relative evidence, re-validate on target profile.

1. EP discovery & session creation

• ov.Core().available_devices → ['CPU'] (full name: Xeon 6985P-C); no NPU/GPU, as expected in this VM.
• ort.get_available_providers() → ['OpenVINOExecutionProvider', 'CPUExecutionProvider'] — the onnxruntime-openvino build exposes the EP correctly.
• Session creation on whisper-base encoder: OpenVINO EP 0.56 s vs CPU EP 0.15 s — OpenVINO compilation adds ~0.4 s cold-start cost per session.

2. whisper-base encoder (ASR-class, 80×3000 features ≈ 30 s audio), warm, best-of-5

Path	Best ms	Mean ms	Peak RSS
ORT CPU EP	160.0	197.9	516 MB
ORT OpenVINO EP (→ OV CPU plugin)	177.7	178.4	475 MB
OpenVINO EP w/ performance_hint=THROUGHPUT	159.0	—	—
Native OpenVINO API (read_model + compile, CPU)	272.7	—	—

• OpenVINO EP is 0.9× (i.e. ~11% slower) than plain CPU EP at best on this model; mean is comparable. No free lunch from the OpenVINO CPU plugin for transformer-encoder workloads on this hardware — ORT's own CPU EP (with its oneDNN-based kernels) is already competitive.
• Numerics: output abs_mean identical to 16 digits between EPs (0.8275253772735596) — bit-comparable.
• Dynamic-shape fragility (important): re-running the same session with a shorter sequence (1000 vs the 3000 it compiled for) fails inside OpenVINO EP (Eltwise shape infer ... mismatch — the EP compiled the graph shape-specialized and cannot re-infer). Plain ORT CPU EP handles dynamic input shapes fine. This mirrors the risk profile of the existing per-shape model exports in the manifest (Trackdub ships fixed-shape variants), but it means the OpenVINO EP would need one session per input-shape bucket — the planner variant machinery already models this.

3. Kokoro-82M (all variants: fp32, fp16, q8f16) — OpenVINO EP cannot load the graph

All three exports fail at session creation:

OpenVINO CPU plug-in doesn't support Parameter operation with dynamic rank.
Operation name: /decoder/decoder/generator/STFT_output_0

The Kokoro graph's STFT output has dynamic rank, which the OpenVINO CPU plugin rejects outright (ORT CPU EP runs the same file fine). So for the current Kokoro TTS artifact, the OpenVINO EP is not a drop-in alternative without a graph fix (pin the STFT output rank/shape in the export).

4. Read-through to the CPU-tier spec (#385)

• No performance case for OpenVINO EP on Intel CPU for the models tested: whisper encoder is slightly slower than the default CPU EP (best case parity with throughput hint), and Kokoro doesn't load at all. The existing ORT CPU EP route (ExecutionProviderKind.Cpu) remains the right CPU default.
• The OpenVINO EP's remaining value on CPU-less-accelerator machines would be: (a) NPU dispatch on Core Ultra hardware (not testable here), (b) its performance_hint throughput mode for batch workloads, (c) potential wins on other model classes (CNV/conv backbones) — none evidenced here.
• The dynamic-rank/dynamic-shape failures reinforce the spec's stance that per-stage EP selection must stay planner-driven with per-variant exports rather than assuming an EP works for every artifact.
• Re-run on the true low-end target profile before drawing final conclusions — this environment has AVX-512 and large caches that may flatter ORT's CPU EP less/more than a laptop part.

Raw results JSON and harness available in the benchmark working notes; can be adapted into Trackdub.Benchmarks provider-matrix runs if wanted.