Phase 0: ASR cassette record/replay layer for GPU-free pipeline tests by pasorobo · Pull Request #1 · pasorobo/WhisperLiveKit

pasorobo · 2026-05-02T09:01:50Z

Summary

Phase 0 of the JA/ZH ASR pipeline plan. Lays the testing foundation needed before adding FireRedASR2 / SenseVoice / language presets in later phases.

Adds a cassette mechanism that captures the interaction between OnlineASRProcessor and an ASR backend at the transcribe → ts_words → segments_end_ts boundary, hash-keyed on the exact audio buffer. The full pipeline (FFmpeg, VAD, online policy, DiffTracker, output formatting) runs for real; only the model inference is served from a JSON file. This unblocks development on Claude Code Web and any other GPU-less environment.

What changed

File	Purpose
`whisperlivekit/test_cassettes.py`	New module: `Cassette`, `CassetteRecorder`, `CassetteASR`, `audio_hash`, schema v1.
`whisperlivekit/core.py` + `config.py`	New `cassette` backend that loads a `CassetteASR` into `TranscriptionEngine`. Routed through `OnlineASRProcessor` (LocalAgreement).
`whisperlivekit/test_harness.py`	`TestHarness.replay(cassette_path, overrides)` and `TestHarness.record(cassette_path, kwargs)` classmethods. Auto-saves cassettes on clean `__aexit__`. Bypasses engine cache in cassette mode to avoid state leaks across tests.
`scripts/record_cassettes.py`	CLI helper that wraps `TestHarness.record()` for batch recording on a GPU machine.
`tests/test_cassettes.py`	10 isolation tests (hashing, token round-trip, recorder, replayer strict/lenient miss, schema validation, core engine integration). All pass with no model required.
`CLAUDE.md`	Tier 1 (replay) / Tier 2 (record) workflow documented.

Tier 1 / Tier 2 workflow

# Tier 2 — GPU machine: record once, commit the cassette JSON
async with TestHarness.record(
    cassette_path="tests/cassettes/qwen3_ja_short_001.json",
    backend="qwen3", lan="ja",
) as h:
    await h.feed("tests/fixtures/ja_short.wav", speed=0)
    await h.finish()

# Tier 1 — Web sandbox / no GPU: replay the cassette
async with TestHarness.replay("tests/cassettes/qwen3_ja_short_001.json") as h:
    await h.feed("tests/fixtures/ja_short.wav", speed=0)
    result = await h.finish()

The cassette is hash-keyed on the exact audio buffer passed to transcribe(), so any change to buffer-trimming or chunking will trigger CassetteMissError on Tier 1 — that's intentional, re-record on the GPU machine when this fires.

Out of scope for Phase 0

Streaming-only backends (Voxtral HF, SimulStreaming) use non-three-tuple result flows and need a future cassette family. v1 covers LocalAgreement-style backends (Whisper, FasterWhisper, MLXWhisper, Qwen3, and the upcoming SenseVoice/FireRedASR2).
No JA/ZH audio fixtures committed yet — those land in Phase 1 alongside the FireRedASR2 / SenseVoice integration when there are real backends to record against.

Test plan

pytest tests/test_cassettes.py -v — 10/10 pass on a GPU-less sandbox
ruff check clean on all new/modified files
python -c "from whisperlivekit.test_harness import TestHarness; assert hasattr(TestHarness, 'replay')" succeeds
Requires GPU machine: run scripts/record_cassettes.py --backend qwen3 --lan ja --audio <wav> --output tests/cassettes/qwen3_ja_smoke.json to validate end-to-end round-trip, then replay it on this branch and confirm pipeline output matches.
After cassettes exist, add a Tier 1 pytest that calls TestHarness.replay() against a committed fixture (deferred to Phase 1).

Generated by Claude Code

Phase 0 of the JA/ZH ASR pipeline plan: introduce a cassette mechanism so the full pipeline (FFmpeg, VAD, online policy, DiffTracker, output formatting) can be exercised end-to-end on a Web sandbox or any GPU-less environment, with model inference served from a JSON file. - whisperlivekit/test_cassettes.py: Cassette / CassetteRecorder / CassetteASR. Captures the (audio, init_prompt) -> (tokens, segment_ends) triples emitted at the OnlineASRProcessor <-> ASR boundary, hash-keyed on the exact audio buffer. - whisperlivekit/core.py + config.py: new "cassette" backend that loads a CassetteASR into TranscriptionEngine. - whisperlivekit/test_harness.py: TestHarness.replay() and .record() classmethods; auto-saves cassette on clean __aexit__; bypasses the engine cache in cassette mode to avoid leaking state. - scripts/record_cassettes.py: CLI for recording cassettes against a real model on a GPU machine. - tests/test_cassettes.py: 10 isolation tests covering hashing, token round-trip, recorder, replayer (strict + lenient miss), schema validation, and the core engine integration. - CLAUDE.md: Tier 1 (replay) / Tier 2 (record) workflow documented. Streaming-only backends (Voxtral HF, SimulStreaming) use a different result flow and need a future cassette family; v1 covers the LocalAgreement-style three-tuple backends only.

Phase 4 of the JA/ZH ASR pipeline plan: Word Error Rate is not a meaningful metric for Japanese, Chinese, or Korean because word boundaries are not orthographically marked. Add Character Error Rate (CER) so committed cassettes and live runs can be scored against ja/zh references. - whisperlivekit/metrics.py: normalize_cjk_text() (NFC, lowercase, strip whitespace + CJK/ASCII punctuation) and compute_cer() with the same character-level Levenshtein DP shape as compute_wer. - whisperlivekit/test_harness.py: TestState.cer() and cer_detailed(). - tests/test_metrics_cjk.py: 19 tests covering Japanese, Chinese, NFC normalisation, mixed CJK/English, punctuation/whitespace insensitivity, and edge cases (empty inputs, CER > 1.0). - CLAUDE.md: documents the cer/cer_detailed methods and explains why word-level WER for ja/zh requires MeCab / jieba (deferred work). This unblocks JA/ZH evaluation for upcoming FireRedASR2 / SenseVoice integrations and for any cassette-based regression test.

Phase 3 of the JA/ZH ASR pipeline plan: pre-bake the recommended backend / language / streaming policy combinations from the report's "use-case cheat sheet" so users get optimal defaults via one flag. - whisperlivekit/presets.py: PRESETS dict with ja-accuracy, ja-realtime, ja-broadcast, zh-accuracy, zh-realtime, ja-zh-en, hri-multilang, apple-silicon-{ja,zh}, en-fast. - whisperlivekit/config.py: WhisperLiveKitConfig.from_preset(name, **overrides). Caller overrides win on conflict. - whisperlivekit/parse_args.py: --preset CLI flag using two-pass argparse so explicit CLI flags naturally override preset defaults. - whisperlivekit/test_data.py: Google FLEURS download for ja_jp and cmn_hans_cn; failures are non-fatal so the test_data flow still works in environments where only some languages are reachable. - tests/test_presets.py: 32 tests covering preset registry hygiene (typo guards via dataclass field validation), every-preset clean loading, override precedence, and CLI integration through parse_args. - CLAUDE.md: presets table; documents Web-sandbox network restrictions (HF and most CDNs blocked, GitHub raw works) so contributors know real-model cassette recording requires a GPU machine with HF access. The zh-* and hri-multilang presets currently target Qwen3 because FireRedASR2 (Phase 1) and SenseVoice (Phase 2) are not yet wired in; they will switch to those backends as those phases land, transparently to anyone using the preset name.

Phases 1 + 2 of the JA/ZH ASR pipeline plan. Closes the two SOTA gaps the report flagged: Chinese (FireRedASR2 averages 2.89% CER on 4 public benchmarks, beating Qwen3-ASR-1.7B / Doubao-ASR / Fun-ASR / Whisper-v3) and a single multilingual backend for HRI use cases (SenseVoice covers zh/en/yue/ja/ko in one non-autoregressive model with emotion + audio- event side-channels). - whisperlivekit/firered_asr.py: FireRedASR2 wrapper around fireredasr2s.fireredasr2.FireRedAsr2. Upstream API takes batches of file paths, so the wrapper writes each numpy chunk to a temp WAV. Auto-resolves AED vs LLM variant from model_size / model_dir; AED produces word-level timestamps, LLM falls back to uniform spacing. - whisperlivekit/sensevoice_asr.py: SenseVoiceASR wrapper around funasr.AutoModel. Parses the <|lang|><|emotion|><|event|><|itn|> tag prefix into ASRToken.detected_language and a metadata dict. Disables the bundled VAD (Silero VAD already runs in the WhisperLiveKit pipeline). Falls back to per-character spacing for CJK without timestamps. - core.py / parse_args.py: register `firered` and `sensevoice` as CLI / config backend choices, both routed through the LocalAgreement policy (no streaming AlignAtt — both models are non-causal AED). - presets.py: zh-accuracy now defaults to FireRed (the actual Mandarin SOTA) instead of Qwen3; hri-multilang now defaults to SenseVoice for the single-model multilingual robot use case. - pyproject.toml: optional [firered] and [sensevoice] extras. The fireredasr2s package has no PyPI release yet so it's pinned to the GitHub repo URL. - tests: 11 isolation tests for FireRed (variant resolution, dict and tuple timestamps, CJK uniform fallback, punctuation segment ends) and 13 for SenseVoice (metadata parsing for ja/en/zh with all four tag classes, ASCII vs CJK token spacing, language propagation). All construct wrappers via __new__ so no model is loaded — Web-safe. End-to-end coverage requires a GPU machine with HF / ModelScope access. The cassette mechanism (Phase 0) covers replay; recording fixtures lands in a follow-up once a contributor runs scripts/record_cassettes.py against real audio.

Phase 5 of the JA/ZH ASR pipeline plan. Surfaces the new public API and makes the use-case cheat sheet from the original report discoverable. - whisperlivekit/__init__.py: re-export FireRedASR2, SenseVoiceASR, the cassette types (Cassette, CassetteASR, CassetteRecorder, CassetteMissError), and the preset helpers (PRESETS, get_preset, list_preset_names). Class import is safe without the optional fireredasr2s / funasr packages installed — model dependencies are loaded lazily inside load_model(). - README.md: new "Japanese / Chinese Backends" section with a use-case → preset → CLI cheat sheet, plus install snippets for the [firered] and [sensevoice] extras. Adds a "Configuration Presets" subsection showing programmatic from_preset() use. - README.md: optional-deps table now lists [firered] and [sensevoice]. - CLAUDE.md: "Adding a New ASR Backend" gains an explicit reference table pointing at firered_asr.py (file-path API), sensevoice_asr.py (metadata tag parsing), and qwen3_asr.py (numpy-in template), so contributors have concrete patterns to copy from for each common backend shape. No new tests — these are documentation and re-export changes only. The 85 existing tests across cassettes, CJK metrics, presets, FireRed, and SenseVoice continue to pass.

Two related fixes for honest real-time semantics: (1) tmpfs-aware temp file in firered_asr.py - _resolve_tmp_root() picks /dev/shm on Linux when available (tmpfs in RAM, ~1-2 ms write vs ~5-10 ms on SSD); falls back to system tmp on macOS, Windows, or any system where /dev/shm is missing or unwritable. - transcribe() now uses tempfile.mkstemp inside that root so each call produces a unique path without us maintaining a counter scheme, and there is no persistent per-session directory to leak. - 3 new tests cover the resolver: prefers /dev/shm on Linux, returns a writable directory on every platform, falls back when /dev/shm is monkeypatched away. (2) Documentation: split backends into "true streaming" vs "quasi-realtime" FireRedASR2 and SenseVoice are non-causal AED; LocalAgreement re-runs full inference on the growing buffer each cycle. That gives ~1-2 s lag on GPU — fine for live captioning and accuracy-first transcription, NOT appropriate for sub-second dialogue UI. The previous wording suggested they were drop-in replacements for Voxtral / SimulStreaming; this commit makes the trade-off explicit: - README.md: new "Streaming Characteristics" table that classifies each backend (✅ true streaming vs ⚠️ quasi-realtime), with concrete latency numbers. Calls out FireRed's per-call temp WAV as a small but non-zero overhead and points users at zh-realtime (Qwen3-SimulKV) when sub-second latency matters more than CER. - CLAUDE.md: Architecture section reorganised into the same two tiers. - presets.py: every preset now carries an inline comment stating whether it targets a true-streaming or quasi-realtime backend, so a contributor reading presets can pick correctly without checking the README. - firered_asr.py / sensevoice_asr.py: top-of-file "Real-time characteristics" section spells out why each is quasi-realtime and when to choose something else. No backend behaviour changes; existing 85 tests + 3 new resolver tests all pass on the Web sandbox.

pasorobo · 2026-05-02T11:55:31Z

Real-time streaming: current pipeline state, limits, and improvement roadmap

Posted as a PR comment because GitHub Issues are disabled on this repository. Treat the checklist below as the tracking artefact for follow-up work after this PR merges.

This documents what works today, what is not truly real-time despite being
wired into the streaming pipeline, and the concrete follow-ups needed to close
those gaps.

Current state (after this PR)

8 ASR backends across two tiers — same pipeline, very different latency
budgets:

Tier	Backend	Mechanism	Typical latency
✅ True streaming	Voxtral Mini Realtime	dedicated streaming `generate` loop	~480 ms
✅ True streaming	Qwen3-MLX-Simul / Qwen3-SimulKV	AlignAtt + KV-cache reuse	~300-500 ms
✅ True streaming	SimulStreaming (Whisper)	AlignAtt + KV-cache reuse	~500 ms
⚠️ Quasi-realtime	Whisper / FasterWhisper / MLXWhisper / Qwen3 (LocalAgreement)	full re-inference on growing buffer	depends on chunk + buffer size
⚠️ Quasi-realtime	FireRedASR2 (this PR)	full re-inference + per-call temp WAV	~1-2 s on GPU
⚠️ Quasi-realtime	SenseVoice (this PR)	full re-inference, numpy in	~1-2 s on GPU

Documentation in README.md ("Streaming Characteristics") and CLAUDE.md
("Architecture") now mirrors this table so users pick correctly.

Concrete limitations

1. LocalAgreement re-runs full inference every cycle

whisperlivekit/local_agreement/online_asr.py:229 calls
self.asr.transcribe(self.audio_buffer, ...) once per cycle. The buffer
grows monotonically up to buffer_trimming_sec (default 15 s) before
being trimmed. For non-causal AED models (Qwen3, FireRed, SenseVoice,
Whisper) this means the same audio is re-encoded many times.

A 1.1 B AED model at RTF 0.087 still needs ~1.3 s to re-encode a
15 s buffer on GPU.
CPU is much worse — typically 5-10× slower.
Increasing min_chunk_size from 0.5 s to 2 s reduces wasted work but
hurts perceived responsiveness.

This is fundamental to LocalAgreement; it cannot be fixed without moving
to a streaming-friendly architecture (KV cache, chunked encoder, or
dedicated streaming decode).

2. FireRedASR2 file-path API forces a temp WAV per call

The upstream fireredasr2s.fireredasr2.FireRedAsr2.transcribe() takes
batches of file paths, not numpy arrays
(upstream API).
firered_asr.py writes each chunk to a temp WAV and calls the model.

Mitigations already in this PR:

_resolve_tmp_root() picks /dev/shm (tmpfs in RAM) on Linux when
available — write cost drops from ~5-10 ms to ~1-2 ms per call.
tempfile.mkstemp per call avoids leaking a session-scoped directory.

Remaining cost: ~10-15 ms per call from sf.write + the model's own
file-decode path. That is ~2-5 % of inference time on GPU but a higher
fraction on CPU.

3. SenseVoice produces no per-word timestamps by default

SenseVoice-Small is non-autoregressive and emits a single
<|lang|><|emotion|><|event|><|itn|>...transcription string. The
2024-11 update added CTC-alignment timestamps but the published
checkpoints don't all expose them. sensevoice_asr.py falls back to
uniform spacing when timestamps are absent — fine for CER scoring but
imprecise for segments_end_ts-driven trim decisions.

4. Tier 2 (real-model) validation has not happened yet

The Web sandbox where this PR was authored cannot reach HuggingFace, so
all new code is verified only against:

isolation tests (24 tests, no model load)
the cassette mechanism (10 tests, synthetic)
the CER metric (19 tests)
preset registry (32 tests)
FireRed tmpfs resolver (3 tests)

Total 88 passing, but no real audio has touched FireRed or
SenseVoice through this pipeline yet. End-to-end correctness
assertions (transcript content, real CER on AISHELL / FLEURS, latency
at streaming defaults) require a GPU machine with HF access.

5. SimulStreaming family does not use the cassette format

The cassette v1 format records the
transcribe → ts_words → segments_end_ts triple at the
OnlineASRProcessor boundary. SimulStreaming, Voxtral HF, and
Qwen3-MLX-Simul use their own non-three-tuple flows — they are wired
through the streaming pipeline directly. This means Tier 1 (Web)
cassette replay tests cannot cover the streaming backends. A
streaming-cassette family (record per-chunk token outputs from a
running generate thread) is needed before we can claim full Tier 1
coverage of the streaming tier.

Improvement opportunities

Ordered roughly by impact / effort:

Investigate FireRed tensor-input entry points. If
fireredasr2.FireRedAsr2._recognize or similar accepts encoder
features / tensors directly, bypass transcribe([uttid], [path])
and skip temp WAVs entirely. Requires reading the upstream code
on a machine with HF access.
Pre-trim with FireRedVAD or Silero VAD before passing audio
to FireRed/SenseVoice. The current pipeline runs Silero VAD but
LocalAgreement still passes the cumulative buffer; an
alternative would be to drop pre-VAD silence so the encoder runs
on shorter input.
Streaming-cassette format (v2) that records per-chunk
streamer outputs for SimulStreaming / Voxtral / Qwen3-MLX-Simul
so Tier 1 tests can cover the true-streaming tier.
Run scripts/record_cassettes.py on a GPU machine with
--backend firered --lan zh and --backend sensevoice --lan ja
against committed FLEURS samples. Commit the resulting JSON
cassettes under tests/cassettes/ so Tier 1 gains real coverage.
Latency benchmark CLI (wlk bench --backend <x> --preset <y>) that measures per-cycle latency p50 / p95 across
backends and writes a CSV. Useful for catching regressions when
buffer-trimming or chunking logic changes.
Investigate SenseVoice CTC alignment to expose per-word
timestamps when the checkpoint supports them — would replace the
uniform-spacing fallback in sensevoice_asr.py:_tokens_from_….
Consider FireRedASR2-LLM streaming via vLLM following the
pattern in vllm_realtime.py (already used for Qwen3). The
upstream vLLM PR (vllm-project/vllm#35727)
may support FireRedASR2-LLM with a streaming generate loop.

Reference

New backend files: whisperlivekit/firered_asr.py,
whisperlivekit/sensevoice_asr.py
Streaming-tier table: README.md ("Streaming Characteristics") and
CLAUDE.md ("Architecture")
Cassette mechanism: whisperlivekit/test_cassettes.py and
scripts/record_cassettes.py

Generated by Claude Code

claude added 6 commits May 2, 2026 09:01

Navigation Menu

Phase 0: ASR cassette record/replay layer for GPU-free pipeline tests - #1

Phase 0: ASR cassette record/replay layer for GPU-free pipeline tests#1
pasorobo wants to merge 6 commits into
mainfrom
claude/design-asr-pipeline-K2QLC

pasorobo commented May 2, 2026

Uh oh!

pasorobo commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

pasorobo commented May 2, 2026

Summary

What changed

Tier 1 / Tier 2 workflow

Out of scope for Phase 0

Test plan

Uh oh!

pasorobo commented May 2, 2026

Real-time streaming: current pipeline state, limits, and improvement roadmap

Current state (after this PR)

Concrete limitations

1. LocalAgreement re-runs full inference every cycle

2. FireRedASR2 file-path API forces a temp WAV per call

3. SenseVoice produces no per-word timestamps by default

4. Tier 2 (real-model) validation has not happened yet

5. SimulStreaming family does not use the cassette format

Improvement opportunities

Reference

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants