Add tts-bench command: Kokoro TTS benchmarking for DubBench

[tonythethompson]

Summary

• Adds a tts-bench command to Trackdub.Benchmarks, adapting the Kokoro-vs-piper Phase 0 harness (evidence in Design spec pitch: piper CPU TTS fallback tier (benchmark-first) #386) into the repo's benchmark tooling for the reference-profile re-run
• New TtsEvalRunner modeled on SeparationEvalRunner: JSONL job list (id, text, language_code, voice_id, optional speed/warmup_runs/repeat_runs), runs the shipped KokoroTtsEngine via HeadlessDubbingHost, warmup + best-of-N repeats, records wall ms (best + mean), audio seconds, RTF, working set before/peak, and provider selection per job as JSONL
• Routes through the engine's own runtime planner (InferenceRequestOptions with model alias / provider pins), so results reflect the real product path, not a synthetic loop; provider pinning via the standard --provider tokens
• Wired into Program command dispatch and BenchmarkConsole usage; 18 new unit tests mirroring SeparationEvalRunnerTests coverage (parse, validation, dispatch, per-job result shape, failure continuation, cancellation)

Usage

Trackdub.Benchmarks tts-bench --jobs jobs.jsonl --results results.jsonl [--model kokoro] [--provider cpu] [--warmup-runs 3] [--repeat-runs 3]

Verification

• dotnet build src/Trackdub.Benchmarks — 0 errors
• dotnet test tests/Trackdub.Benchmarks.Tests --filter FullyQualifiedName~TtsEvalRunnerTests — 18/18 passed
• Full project run: 532 passed, 2 pre-existing failures in ResourceTelemetryOptionsTests culture tests (reproduced on clean tree without this change — sandbox lacks ICU libraries, forcing invariant globalization; not related to this PR)

[cursor]

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• queue - adds this PR to the back of the merge queue
• fast - for urgent changes, fast-track this PR to the front of the merge queue

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 7019b475-fd9d-40bb-8838-e7537f06b003

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[amazon-q-developer]

This PR successfully integrates a TTS benchmarking command (tts-bench) into the Trackdub.Benchmarks tooling. The implementation follows established patterns from SeparationEvalRunner, includes comprehensive test coverage (18 tests), and properly routes through the real product path via HeadlessDubbingHost and KokoroTtsEngine.

Key strengths:

• Clean separation of concerns with dedicated records for jobs, options, and results
• Robust error handling with graceful failure continuation
• Proper resource management with WorkingSetPeakMonitor and disposal patterns
• Comprehensive test coverage mirroring existing benchmark test patterns
• JSONL streaming output for incremental result capture

The code is well-tested (18/18 tests passing), follows the repository's conventions, and integrates cleanly with the existing benchmark infrastructure. No blocking issues identified.

---

You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

[tonythethompson]

Two things that affect whether the numbers mean what the PR says they mean; inline below.

[Copilot]

Copilot review overview

🟡 Changes recommended

Model pinning, alias handling, per-job validation, and provider usage text can currently produce incorrect or unusable benchmarks.

Review effort: Balanced
Findings: 3

Open (3)

• Display valid execution-provider tokens in usage text · New
• Validate per-job repeat and warmup overrides · New
• Resolve Kokoro adapter independently from model alias · New

What changed in this PR

Adds a Kokoro TTS benchmarking command for collecting timing, RTF, memory, and provider evidence.

Changes:

• Adds JSONL-driven TTS benchmark execution and reporting.
• Registers tts-bench and adds CLI usage.
• Adds tests and the related Piper benchmark-first design spec.

File	Description
src/​Trackdub.Benchmarks/​TtsEvalRunner.cs	Implements parsing, execution, and JSONL results.
src/​Trackdub.Benchmarks/​Program.cs	Dispatches the new command.
src/​Trackdub.Benchmarks/​BenchmarkConsole.cs	Documents command usage.
tests/​Trackdub.Benchmarks.Tests/​TtsEvalRunnerTests.cs	Tests parsing and execution behavior.
docs/​specs/​design-piper-cpu-tts-tier.md	Documents the benchmark-first Piper proposal.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tonythethompson]

OpenVINO EP vs CPU EP smoke results (Intel CPU sandbox)

Environment: same constrained-CPU sandbox as the TTS benchmark — Intel Xeon 6985P-C @ 2.30 GHz, no GPU/NPU. Stack: openvino 2026.4.1, onnxruntime-openvino 1.24.1 (ORT with OpenVINO EP + CPU EP). Caveat as before: server-class constrained environment, not a low-end laptop; relative evidence, re-validate on target profile.

1. EP discovery & session creation

• ov.Core().available_devices → ['CPU'] (full name: Xeon 6985P-C); no NPU/GPU, as expected in this VM.
• ort.get_available_providers() → ['OpenVINOExecutionProvider', 'CPUExecutionProvider'] — the onnxruntime-openvino build exposes the EP correctly.
• Session creation on whisper-base encoder: OpenVINO EP 0.56 s vs CPU EP 0.15 s — OpenVINO compilation adds ~0.4 s cold-start cost per session.

2. whisper-base encoder (ASR-class, 80×3000 features ≈ 30 s audio), warm, best-of-5

Path	Best ms	Mean ms	Peak RSS
ORT CPU EP	160.0	197.9	516 MB
ORT OpenVINO EP (→ OV CPU plugin)	177.7	178.4	475 MB
OpenVINO EP w/ performance_hint=THROUGHPUT	159.0	—	—
Native OpenVINO API (read_model + compile, CPU)	272.7	—	—

• OpenVINO EP is 0.9× (i.e. ~11% slower) than plain CPU EP at best on this model; mean is comparable. No free lunch from the OpenVINO CPU plugin for transformer-encoder workloads on this hardware — ORT's own CPU EP (with its oneDNN-based kernels) is already competitive.
• Numerics: output abs_mean identical to 16 digits between EPs (0.8275253772735596) — bit-comparable.
• Dynamic-shape fragility (important): re-running the same session with a shorter sequence (1000 vs the 3000 it compiled for) fails inside OpenVINO EP (Eltwise shape infer ... mismatch — the EP compiled the graph shape-specialized and cannot re-infer). Plain ORT CPU EP handles dynamic input shapes fine. This mirrors the risk profile of the existing per-shape model exports in the manifest (Trackdub ships fixed-shape variants), but it means the OpenVINO EP would need one session per input-shape bucket — the planner variant machinery already models this.

3. Kokoro-82M (all variants: fp32, fp16, q8f16) — OpenVINO EP cannot load the graph

All three exports fail at session creation:

OpenVINO CPU plug-in doesn't support Parameter operation with dynamic rank.
Operation name: /decoder/decoder/generator/STFT_output_0

The Kokoro graph's STFT output has dynamic rank, which the OpenVINO CPU plugin rejects outright (ORT CPU EP runs the same file fine). So for the current Kokoro TTS artifact, the OpenVINO EP is not a drop-in alternative without a graph fix (pin the STFT output rank/shape in the export).

4. Read-through to the CPU-tier spec (#385)

• No performance case for OpenVINO EP on Intel CPU for the models tested: whisper encoder is slightly slower than the default CPU EP (best case parity with throughput hint), and Kokoro doesn't load at all. The existing ORT CPU EP route (ExecutionProviderKind.Cpu) remains the right CPU default.
• The OpenVINO EP's remaining value on CPU-less-accelerator machines would be: (a) NPU dispatch on Core Ultra hardware (not testable here), (b) its performance_hint throughput mode for batch workloads, (c) potential wins on other model classes (CNV/conv backbones) — none evidenced here.
• The dynamic-rank/dynamic-shape failures reinforce the spec's stance that per-stage EP selection must stay planner-driven with per-variant exports rather than assuming an EP works for every artifact.
• Re-run on the true low-end target profile before drawing final conclusions — this environment has AVX-512 and large caches that may flatter ORT's CPU EP less/more than a laptop part.

Raw results JSON and harness available in the benchmark working notes; can be adapted into Trackdub.Benchmarks provider-matrix runs if wanted.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T16:29:02.172753Z	6a641db	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[qodo-code-review]

PR Summary by Qodo

Add Kokoro TTS benchmark command and benchmark-first Piper spec

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Add JSONL-driven Kokoro benchmarks through the shipped engine to measure CPU performance and
 provider selection.
• Record per-job timing, real-time factor, memory usage, and failures for reference-profile
 comparisons.
• Document evidence gates for a possible Piper fallback and test command parsing and execution.

Diagram

graph TD
  CLI["tts-bench command"] --> Runner["TTS eval runner"] --> Host["Headless host"] --> Engine["Kokoro adapter"] --> Planner["Runtime planner"]
  Jobs[("Jobs JSONL")] --> Runner
  Runner --> Results[("Results JSONL")]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Benchmark through RoutedTtsEngine

• ➕ Exercises product-level adapter selection and future fallback behavior.
• ➖ Planner-selected engines could undermine fixed-engine performance comparisons.

Recommendation: Keep the direct Kokoro adapter path for controlled CPU comparisons: its synthesis method still uses the runtime planner for model and provider selection. If the later study needs to measure Kokoro-to-Piper fallback behavior, add a separate routed benchmark mode rather than changing this baseline.

Files changed (5) +731 / -0

Enhancement (3) +438 / -0

BenchmarkConsole.csShow tts-bench in command usage +2/-0

Show tts-bench in command usage

• Adds the TTS benchmark command and its principal options to top-level help.

src/Trackdub.Benchmarks/BenchmarkConsole.cs

Program.csDispatch the tts-bench command +28/-0

Dispatch the tts-bench command

• Registers the command and handles option errors, help, and runner exit codes.

src/Trackdub.Benchmarks/Program.cs

TtsEvalRunner.csRun JSONL TTS jobs and capture performance results +408/-0

Run JSONL TTS jobs and capture performance results

• Parses jobs and options, resolves a TTS adapter from a headless host, and performs warmups and repeated synthesis. Writes per-job timing, real-time factor, working set, provider details, and failures to JSONL.

src/Trackdub.Benchmarks/TtsEvalRunner.cs

Tests (1) +206 / -0

TtsEvalRunnerTests.csCover TTS job parsing, dispatch, metrics, and failures +206/-0

Cover TTS job parsing, dispatch, metrics, and failures

• Tests argument and JSONL validation, help dispatch, request options and result fields, failure continuation, and cancellation using a fake engine.

tests/Trackdub.Benchmarks.Tests/TtsEvalRunnerTests.cs

Documentation (1) +87 / -0

design-piper-cpu-tts-tier.mdPropose evidence-gated Piper CPU fallback +87/-0

Propose evidence-gated Piper CPU fallback

• Defines reference-hardware benchmarks and eSpeak-NG failure investigation as prerequisites for a Piper tier. Specifies planner-owned fallback selection, packaging and readiness checks, and licensing gates if that tier proceeds.

docs/specs/design-piper-cpu-tts-tier.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Jobs with no measured runs pass ✗ Dismissed

Description

ReadJobs accepts a per-job repeat_runs value of zero even though the command-line equivalent
requires a positive value. When warmups are also zero, RunJobAsync performs no synthesis but
records the job as successful with zero timing and no audio.

Code

src/Trackdub.Benchmarks/TtsEvalRunner.cs[R320-321]

+        int warmupRuns = job.WarmupRuns ?? options.WarmupRuns;
+        int repeatRuns = job.RepeatRuns ?? options.RepeatRuns;

Relevance

●●● Strong

This is a deterministic validation gap: per-job repeat overrides should enforce the same
positive-count invariant as global options.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The global repeat count is validated, but job parsing checks only required fields. The job override
controls a loop that runs zero times for repeat_runs: 0, after which the method constructs an `Ok:
true` result.

src/Trackdub.Benchmarks/TtsEvalRunner.cs[89-101]
src/Trackdub.Benchmarks/TtsEvalRunner.cs[154-175]
src/Trackdub.Benchmarks/TtsEvalRunner.cs[320-357]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A JSONL job can set `repeat_runs` to zero and produce a successful record without a measured synthesis.
## Fix Focus Areas
- src/Trackdub.Benchmarks/TtsEvalRunner.cs[163-175]
- src/Trackdub.Benchmarks/TtsEvalRunner.cs[320-321]
## Recommended Fix
Reject per-job `repeat_runs` below one and `warmup_runs` below zero during job parsing; add tests for both overrides.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Help suggests providers the command rejects ✓ Resolved

Description

BenchmarkConsole.WriteUsage fills the --provider choices with Windows ML device-policy keys
rather than execution-provider tokens. A user following its advertised explicit or
max-performance value reaches TtsEvalOptions.TryParse, which rejects the value as an unknown
provider.

Code

src/Trackdub.Benchmarks/BenchmarkConsole.cs[23]

+        writer.WriteLine($"  Trackdub.Benchmarks tts-bench --jobs <jobs.jsonl> --results <results.jsonl> [--model <alias>] [--provider cpu|{WindowsMlExecutionDevicePolicySettings.FormatSupportedKeys("|")}] [--warmup-runs <n>] [--repeat-runs <n>]");

Relevance

●●● Strong

Accepted provider-alias mismatch fixes show this team prioritizes consistent CLI tokens and runtime
parsing.

PR-#255

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The help interpolates device-policy keys such as explicit and max-performance. The parser
instead uses ExecutionProviderTokens.TryParse, whose provider map contains values such as cpu
and dml, not those policy keys.

src/Trackdub.Benchmarks/BenchmarkConsole.cs[23-23]
src/Trackdub.Contracts/ApplicationContracts/WindowsMlEpDevicePolicyContracts.cs[32-53]
src/Trackdub.Domain/Common/ExecutionProviderTokens.cs[18-48]
src/Trackdub.Benchmarks/TtsEvalRunner.cs[113-118]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The command's help lists device policies as provider values, but the parser accepts execution-provider tokens instead.
## Fix Focus Areas
- src/Trackdub.Benchmarks/BenchmarkConsole.cs[23-23]
## Recommended Fix
Generate the provider choices from `ExecutionProviderTokens` and add a usage test that checks the displayed values against the parser.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Valid model aliases cannot be benchmarked ✓ Resolved

Description

RunAsync compares --model with an adapter's engine family before passing the value to the
runtime planner. With the manifest-supported alias kokoro-onnx, that comparison finds no adapter
and exits without running any jobs.

Code

src/Trackdub.Benchmarks/TtsEvalRunner.cs[223]

+                .FirstOrDefault(e => string.Equals(e.EngineFamily, options.Model, StringComparison.OrdinalIgnoreCase));

Relevance

●●● Strong

Accepted model-validation precedents support resolving aliases against runtime-backed model
capabilities, not merely engine-family names.

PR-#257

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The manifest assigns both kokoro-onnx and kokoro to the kokoro engine family. The runner
selects adapters by exact family comparison, while Kokoro passes the requested alias to the planner
only after adapter selection.

src/Trackdub.Inference/Runtime/ModelManifest/bundled-models.manifest.json[2308-2336]
src/Trackdub.Benchmarks/TtsEvalRunner.cs[221-228]
src/Trackdub.Inference.Onnx/Kokoro/KokoroTtsEngine.cs[78-101]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`--model kokoro-onnx` is a valid manifest alias but is rejected because adapter lookup compares it with the engine family.
## Fix Focus Areas
- src/Trackdub.Benchmarks/TtsEvalRunner.cs[221-227]
## Recommended Fix
Resolve the requested alias through the model manifest to select its engine family, then pass the original alias to the engine and planner.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. A memory sample can stop all jobs ✗ Dismissed

Description

RunJobAsync calls CaptureWorkingSetBytes before entering its per-job exception handler. If that
initial sample throws, no failed result is written for the job and execution never advances to the
remaining jobs.

Code

src/Trackdub.Benchmarks/TtsEvalRunner.cs[309]

+        long before = sampler.CaptureWorkingSetBytes();

Relevance

●●● Strong

Accepted benchmark telemetry fixes favor containing sampler failures and preserving job continuation
or unavailable telemetry.

PR-#275
PR-#196

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The initial capture precedes the try, whereas the peak monitor explicitly treats sampler
exceptions as unavailable telemetry. RunJobsAsync awaits each job sequentially, so an escaped
exception prevents later result lines.

src/Trackdub.Benchmarks/TtsEvalRunner.cs[288-296]
src/Trackdub.Benchmarks/TtsEvalRunner.cs[309-322]
src/Trackdub.Benchmarks/WorkingSetPeakMonitor.cs[75-108]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
An exception from the initial working-set sample escapes the per-job handler and aborts the benchmark.
## Fix Focus Areas
- src/Trackdub.Benchmarks/TtsEvalRunner.cs[309-322]
## Recommended Fix
Move the initial sample inside the per-job try block and handle sampler failures without aborting subsequent jobs, consistent with the working-set monitor's best-effort behavior.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 4 rules

✅ Cross-repo context — repo relationships

✅ REVIEW.md

Review mode: Auto: ⚖️ Balanced: Downgraded extended -> standard: change is below the extended eligibility bar (hunks 6/18, lines 731/200; both must reach the floor). Router rationale: New benchmark runtime, CLI integration, measurement logic, and failure paths create multiple defect opportunities.

Tip of the day

💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a641db799

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[qodo-code-review]

Qodo Fixer

🍒 Ready to be cherry-picked — ✅ Merged (0) · ☑ Fixed (1)

🔗 Fix PR: #389

This fix PR was closed automatically. Its branch is preserved so you can cherry pick the changes into the original PR.

Prompt for coding agent

This is an automated fix prepared on a separate branch (#389). It is NOT applied to this PR.
To use it: review Fix PR #389 (https://github.com/trackdubllc/Trackdub/pull/389), evaluate each change critically against your local context, and cherry-pick the changes that are correct into this branch. Do not accept them blindly.

Process — 1 fixed

• ☑ Fixed: Jobs with no measured runs pass

[cubic-dev-ai]

All reported issues were addressed across 5 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[tonythethompson]

/oc fix

[opencode-agent]

Automated review: 5 inline finding(s); the summary follows as a comment.

[opencode-agent]

Review posted successfully. Here is the summary:

Review submitted — 5 inline findings across 2 files.

Overall: The tts-bench command is well-structured with good test coverage, but two correctness/reliability gaps in per-job validation and working-set sampling can produce misleading benchmarks or abort the entire run. Three smaller findings address debuggability and spec accuracy.

#	Severity	File:Line	Finding
1	important	src/Trackdub.Benchmarks/TtsEvalRunner.cs:170	Per-job warmup_runs/repeat_runs overrides bypass CLI validation — repeat_runs: 0 produces Ok: true with a fabricated Rtf: 0
2	important	src/Trackdub.Benchmarks/TtsEvalRunner.cs:309	Initial working-set sample is outside the per-job try block — a sampler exception aborts all remaining jobs
3	suggestion	src/Trackdub.Benchmarks/TtsEvalRunner.cs:390	CreateFailedResult hardcodes RequestedProvider: null — failed rows lose the --provider pin
4	suggestion	docs/specs/design-piper-cpu-tts-tier.md:42	Spec references espeakng-loader which doesn't exist — should be EspeakNgPhonemizer
5	suggestion	docs/specs/design-piper-cpu-tts-tier.md:56	Spec cites design-llamacpp-cpu-text-tier.md which doesn't exist in the repo

Out of diff: No out-of-diff findings. All 5 findings are anchored inline on the PR diff.

github run

[opencode-agent]

The new commit isn't pushed yet (the action pushes automatically); the last-pushed commit e0bc6e1 had all checks green, and my post-merge build + tests are green locally. The pass is complete.

Summary

Conflicts — None. Merged latest origin/main (fbf42fb→ddb1dc5, includes #374/#376/#378/#380/#382/#393) into the head branch; clean, no conflicts, and Program.cs changes on main don't affect the tts-bench dispatch.

Fixed (all 21 review threads resolved):

• Require the Kokoro alias (tonythethompson, chatgpt-codex-connector, cubic-dev-ai): RunAsync now resolves RoutedTtsEngine and the request sets RequirePreferredModelAlias: true, matching the product dubbing path so the planner can't silently land on a different TTS model. --model kokoro-onnx now flows to the planner as a required alias (adapter lookup keyed on engine family, not the model string). src/Trackdub.Benchmarks/TtsEvalRunner.cs:243, :333.
• Load/warmup peak (tonythethompson): a WorkingSetPeakMonitor now starts before the warmup loop and writes warmup_peak_working_set_bytes, capturing session creation + model load alongside the steady-state peak. TtsEvalRunner.cs:342-354.
• Per-job override validation (copilot, qodo, codex, cubic): ReadJobs rejects warmup_runs < 0 and repeat_runs < 1 so repeat_runs: 0 can no longer produce a fabricated Ok result. TtsEvalRunner.cs:171-180.
• Sampler failure containment (qodo, cubic): initial working-set capture moved inside the per-job try via CaptureWorkingSetBestEffort, so a throwing sampler records unavailable telemetry instead of aborting all jobs. TtsEvalRunner.cs:342, :402.
• Failed rows keep the provider pin (cubic): CreateFailedResult now receives options.Provider and preserves it as RequestedProvider. TtsEvalRunner.cs:393, :416-434.
• Spec doc accuracy (cubic): espeakng-loader → EspeakNgPhonemizer; removed the nonexistent design-llamacpp-cpu-text-tier.md citation (§4, §8); Phase 0 now includes the native-listener intelligibility review matching its stated goal. docs/specs/design-piper-cpu-tts-tier.md.

Tests — Updated sampler sequence for the warmup monitor; added ReadJobs_RejectsInvalidPerJobRunOverrides, RunJobsAsync_SamplerFailureDoesNotAbortRemainingJobs, plus RequirePreferredModelAlias, warmup_peak_working_set_bytes, and failed-row-provider assertions. 697/698 pass in Trackdub.Benchmarks.Tests; Release -warnaserror build clean.

Not fixed (resolved as not valid) — cubic-dev-ai Program.cs refactor suggestion declined: RunTtsBenchAsync and RunSeparationEvalAsync have different error contracts (TtsEvalRunner.RunAsync owns its exception translation; RunSeparationEval delegates it to the command layer), so a shared scaffold would obscure the difference. Replied and resolved.

CI — No failures to fix; prior head commit was fully green, and the current merge + changes build and test clean locally. Checks on the new commit will run on push.

github run

[tonythethompson]

/oc fix merge conflicts and failing ci tests

[opencode-agent]

All conflicts resolved, CI fix verified, working tree clean. Final state check done.

Summary

Conflicts — resolved. Merged the latest origin/main (a13a802 + intervening commits: f4ddc08, 5b04a96, a020342, 90cdcd7, cabe2ab, bdee843, b4e5fa3) into the head branch. Two conflicts, both in the per-job warmup_runs/repeat_runs override validation that main had independently added (same fix as this PR, from the closed Qodo fixer PR #389):

• src/Trackdub.Benchmarks/TtsEvalRunner.cs:171-186 — both sides add identical validation with slightly different error wording; kept this PR's version (repeat_runs must be a positive integer).
• tests/Trackdub.Benchmarks.Tests/TtsEvalRunnerTests.cs:41-49 — kept this PR's richer theory (3 InlineData cases incl. repeat_runs:-3).
• Merge commit 3b55c19; no markers left, working tree clean, prospective merge vs origin/main conflict-free.

CI — fixed. Build & Test (Linux/Windows/macOS) all failed on InferenceSessionPoolTests.ProcessGpuAdmission_ProcessAtDeviceCeiling_FailsFastWithNothingIdleLeftToFree. This is a pre-existing failure on main itself (main's Build & Test (Linux) on a13a802 fails identically), inherited by this PR — the PR does not touch Inference. Root cause: #376 added the test asserting the diagnostic phrase "dedicated GPU memory outside", then #378 rewrote the pool's message to "dedicated GPU memory is attributed to this bucket" without updating the assertion. Updated the stale assertion at tests/Trackdub.Inference.Tests/InferenceSessionPoolTests.cs:939.

Verification:

• dotnet build Trackdub.slnx --configuration Release -m:1 -warnaserror — 0 warnings, 0 errors
• Trackdub.Benchmarks.Tests — 697 passed / 1 skipped (22 TtsEvalRunnerTests pass)
• Trackdub.Inference.Tests — 1067 passed / 46 skipped; previously failing test now passes

Threads — all 28 review threads are resolved; no new unresolved feedback to triage (last-pass summary comment is the only timeline comment besides the trigger).

Checks on the new commit (3b55c19) will run on push; merge conflict is cleared.

github run