Voice Agent

An offline, low-latency local voice agent with real turn-taking. Talk to it, talk over it, and it behaves the way a person does: it stops, listens, works out whether you actually wanted the floor, and either picks up where it left off or answers what you just asked.

Runs on a laptop microphone or answers a phone call over Twilio. Everything is local — no audio leaves the machine except on the telephony path.

• Rust core (voice_core) — PyO3 voice activity detection.
• STT — faster-whisper (CTranslate2) or mlx-whisper.
• LLM — any MLX chat model; defaults to Qwen 2.5 3B Instruct 4-bit on Metal.
• TTS — Kokoro v0.19 ONNX at 24kHz, macOS system voice as fallback.

Quick start

Requires macOS on Apple Silicon (the LLM runs on MLX/Metal) and Python 3.13.

git clone https://github.com/ml-lubich/voice-agent
cd voice-agent
pip install -e .
voice-agent

That is the whole setup. On first run it downloads the 310MB Kokoro voice model into models/ — once, automatically. Everything after that is fully offline.

Prefer uv? uv sync && uv run voice-agent works too; the project is a standard PEP 621 package with a hatchling backend, so pip, uv, pipx and Poetry all handle it.

Commands

voice-agent                  # live microphone (default)
voice-agent --adapt          # ...and size the models to this machine
voice-agent text             # type instead of talk
voice-agent serve            # answer calls over Twilio
voice-agent bench            # measure the latency budget on your hardware
voice-agent config           # print the effective config
voice-agent config --save    # write ~/.config/voice-agent/config.toml

Development

pip install -e ".[dev]"      # adds pytest + ruff
pytest                       # 42 fast tests, ~25s, no models needed
pytest -m integration        # real models and real audio, slower
ruff check src tests

There is a Makefile if you prefer: make dev, make test, make test-all, make bench, make lint, make rust.

Optional: the Rust extension

Not required — the agent runs on numpy fallbacks without it, and the tests check the Rust against those fallbacks. Build it to take the audio maths off the critical path (see Rust core):

pip install -e ".[rust]" && maturin develop --release

How it compares

Most open-source voice agents are orchestrators: you bring API keys, they manage the turn-taking and stream audio between hosted STT/LLM/TTS. This one is a self-contained local agent that happens to also answer the phone.

	this	Pipecat	LiveKit Agents	Vapi / Retell
Runs fully offline	yes	with local plugins	no (media plane is hosted)	no
Transport	mic + Twilio	many	LiveKit WebRTC	hosted
Turn-taking policy	pure, exhaustively tested	in-pipeline processors	model-based + VAD	hosted, opaque
Backchannel handling	duck-then-decide, local	configurable	trained classifier	hosted
Dependencies to run	Python + one Rust ext	framework + services	framework + cloud	account

I have not benchmarked the others on this hardware, so there is no honest latency column here. Their numbers are dominated by network round-trips to hosted models, which is a different problem to the one this solves; comparing my local p50 against their cloud p50 would be measuring the network, not the code. What is comparable is the architecture, and the one genuine difference is that the turn-taking policy here is a pure function you can run 72,000 conversations through in 15 seconds — see Testing.

LiveKit's adaptive-interruption work is the closest published equivalent to the duck-then-decide behavior below; theirs uses a trained acoustic classifier, this uses a threshold plus a transcript check, which is cheaper and explainable but will be worse on genuinely ambiguous audio.

Latency

Time to first audio is the number that matters — playback of the first phrase covers synthesis of the rest, so total roundtrip is a vanity metric.

Measured on an Apple M3 Max, 8 turns, speech synthesized in a different voice so STT does real work (voice-agent bench):

	p50	p95	worst	within 800ms
default (stt.model=tiny)	643ms	747ms	787ms	8/8
stt.model=base	779ms	876ms	999ms	5/8
Qwen3-4B instead of 2.5-3B	1066ms	1265ms	1329ms	1/8

Budget breakdown per turn: ~130ms STT, ~160ms to the LLM's first tokens, ~250-380ms Kokoro synthesis. Kokoro has a fixed ~130ms phonemization cost, which is why the first chunk is cut short — synthesis time scales with length, so a short opening phrase is cheaper and arrives sooner.

Not verified on an M4 Mac mini. The M3 Max has considerably more GPU cores; expect the LLM stage to be slower there. voice-agent bench reports the real number on any machine.

Things that did not work, measured rather than assumed:

• int8 STT is slower than float32 (305ms vs 221ms). CTranslate2's float32 path gets Accelerate/AMX on Apple Silicon; int8 does not.
• CoreML for Kokoro is not faster. ONNX splits the graph into 109 partitions (1044 of 2262 nodes supported) and the partition overhead eats the gain.
• Qwen3-4B is slower and no better for one-or-two-sentence spoken replies. It stays a one-flag switch: --set llm.model=....

Rust core

The per-frame audio maths lives in crates/voice_core (PyO3, zero-copy numpy — no Python-list conversion at the boundary). Measured against the numpy reference on an M3 Max, min-of-N:

	Rust	numpy
rms (512-sample frame)	0.47µs	2.16µs	4.6x
resample (48k samples)	28.7µs	326µs	11.4x
ulaw_encode (48k samples)	98.8µs	194µs	2.0x
ulaw_decode (48k samples)	243µs	246µs	1.0x — no win

ulaw_decode is a wash: numpy indexing a 256-entry lookup table is already optimal, and it is listed here rather than quietly dropped.

The VAD frame path costs 0.48µs per frame, or 0.0015% of one core at 32ms frames — which is the point for low-compute devices: the audio plumbing disappears from the budget entirely and everything left is model time.

The extension is optional. The numpy implementations are the reference, the tests check Rust against them, and the agent runs unmodified if the extension was never built — just with a larger constant factor. Build it with uv run maturin develop --release.

Note the default resampler is scipy's polyphase, not the faster Rust linear one: the 24kHz→8kHz phone path is a 3:1 decimation and needs an anti-aliasing filter. Rate conversion runs once per TTS chunk, not once per frame, so correctness wins. resample_linear is exposed for the non-decimating case.

Adapting to the machine

--adapt probes cores, memory and whether there is a usable accelerator, then picks model sizes for that tier. It sits below config files, environment variables and flags, so it can never silently undo something you set.

tier
workstation	accelerated, ≥24GB, ≥10 cores	3B model, 100 tokens
laptop	accelerated, ≥12GB	3B model, 80 tokens
server	≥8 cores, ≥8GB, no accelerator	1.5B model, 70 tokens
edge	anything smaller	1.5B, longer first chunk, wider VAD window, later filler

The edge profile deliberately makes the first chunk longer. Kokoro has a fixed per-call cost, so on a slow machine paying it fewer times beats starting marginally sooner.

Turn-taking

The policy lives in turn.py and is deliberately pure — no threads, no audio devices, no models, no wall clock. Three separate decisions, because conflating them is the bug:

1. VAD — is this frame speech?
2. Endpointing — has the user finished their turn?
3. Interruption handling — what do we do about speech that arrives while the agent is talking?

Barge-in: duck, then decide

Pausing is instant and reversible, so the agent is eager to pause and slow to commit:

• Sustained speech over the agent (min_duration_ms) pauses playback.
• The interruption is transcribed. A backchannel — "mhm", "okay", "right" — means keep going: playback resumes where it stopped.
• Anything substantive cancels the reply properly: playback stops, generation is abandoned, queued audio is dropped, and the history records that the rest was never heard. Skipping any one of those four is its own bug.
• Speech sustained past commit_ms is a real interruption regardless of what the transcript says. Half a second of talking is not a backchannel.

Not getting stuck

• echo_gate_ms — the agent is deaf for a beat after its own audio starts, so the leading edge of its own voice can't trigger a barge-in. Without echo cancellation this is what stops the self-interruption loop.
• max_duck_ms — if the mic is hot but nothing turn-shaped arrives (a fan, a TV, speaker bleed), resume and re-arm the echo gate.
• max_interruption_ms — an absolute ceiling. Continuous noise above the barge-in threshold never endpoints, so at this point take what we have as the turn rather than talk over someone who may genuinely be speaking.
• Silence, punctuation-only transcripts and Whisper's near-silence hallucinations ("you", "Thanks for watching!") never become turns.
• History is bounded, so latency doesn't creep upward over a long conversation.

Testing

Turn-taking bugs are timing bugs, so nothing here depends on real time.

• 72,000 generated scenarios (tests/test_turn_sweep.py) — every combination of 5 tuning profiles × 3 starting states × onset × duration × level × 4 noise environments, replayed frame by frame on a virtual clock in ~15s. Each is checked against the invariants: never duck inside the echo gate, never turn a cough into a turn, never stay ducked past the ceiling, resolve every duck exactly once.
• End-to-end conversations (tests/test_session.py) — a mocked agent and a scripted microphone, virtual clock, no sleeps and no flakes.
• Threaded conversations (tests/test_threaded.py) — the same scenarios through the real worker-thread path, paced at 8x realtime. sync=True hides an entire class of bug: inline workers mean a reply's history entry always exists before anything inspects it, and no cancellation can land mid-stream.
• Real audio (tests/test_real_audio.py, -m integration) — real Whisper, real LLM, real Kokoro, with the "human" synthesized in a different voice.

Both extra layers earned their keep immediately.

The real-audio tests caught that cancelling a reply bumped the generation counter the replacement reply was checked against, so the agent went silent exactly when you'd interrupted to ask something else. The mocked tests missed it because they only asserted the interruption was cancelled, not that the new question got answered.

A fresh-context review then found what all of the above still missed, because every test ran sync=True:

• mark_interrupted() was a no-op on the real barge-in path. Cancellation arrives while the reply generator is still suspended, so the last history entry is the user message, not the assistant one it was sniffing for. The model went on believing it had said the whole reply.
• A duck could pause the transport with nothing left to un-pause it, swallowing an entire reply in silence with no error anywhere.
• was_ducked was inferred from pending_ms > 0, which is wrong in both directions.
• The echo gate was armed when the agent decided to reply, ~400ms before its first audible sample — so the anti-self-trigger guard had always expired by the time it was needed.
• A cough left _capturing set forever, so STT was eventually handed minutes of audio.

tests/test_threaded.py exists because of that review, and pins each one.

Configuration

Everything is persistent and layered — later wins:

1. defaults in config.py
2. ~/.config/voice-agent/config.toml
3. ./voice-agent.toml
4. VOICE_AGENT__BARGE_IN__THRESHOLD=0.08
5. --set barge_in.threshold=0.08 (repeatable)

uv run voice-agent config --save          # write the user config file
uv run voice-agent config --get vad.silence_ms
uv run voice-agent --set stt.model=base --set vad.silence_ms=450 voice

The knobs that matter in a new room:

key	default
vad.threshold	0.015	speech vs. room
vad.silence_ms	700	how long a pause ends your turn
barge_in.threshold	0.05	how loud you must be to cut in
barge_in.echo_gate_ms	250	self-interruption guard
barge_in.max_duck_ms	1200	give up on a false interruption
barge_in.commit_ms	600	past here, never resume
stt.model	tiny	base/small for accuracy over latency

Headphones remove the echo problem entirely; on speakers, raise barge_in.threshold if the agent interrupts itself and lower it if you can't cut in.

Twilio

voice-agent serve opens a Media Streams websocket (8kHz μ-law, 20ms frames). Point a number's voice webhook at a URL returning:

<Response><Connect><Stream url="wss://your-host/stream" /></Connect></Response>

Barge-in sends Twilio's clear message, which is the only way to drop audio Twilio has already buffered for the caller — without it the abandoned reply keeps playing in their ear for seconds after the agent stopped sending it.

G.711 μ-law is implemented in transport.py with numpy (~37dB SNR): Python 3.13 removed audioop, and the codec is a dozen lines.