A macOS desktop widget that turns Claude into a cute voice assistant. pocket-tts is the voice, Moonshine is the ears, and both run locally on CPU. The mic stays on: talk or type, and it answers out loud. Claude runs through your existing claude CLI and Pro/Max subscription, no API key needed.
pocketpal_short_demo.mp4
If you find this interesting, you can also check out the following project:
- Unmute by Kyutai: The initial inspiration. Open-source "always-on" STT+TTS wrapper using more advanced voice models
- Gradbot by Gradium: Open-source framework to build a custom real-time voice agent in minutes
- Moonshine: An open-source framework for building real-time voice agents
Docs available at: https://ameroyer.github.io/pocketpal/
Start a Claude session in a tmux pane (either locally or on a remote host reachable via SSH). Set it to auto-mode if you want to be able to chat without being prompted/interrupted by approval requests in the Claude session terminal.
# Open PocketPal (no clone needed)
uvx git+https://github.com/ameroyer/pocketpal
# or, from a clone of this repo
uv run python -m pocketpalOr double-click PocketPal.command in Finder (first time: right-click → Open, since it isn't signed). It installs uv if needed. The first run downloads the models (a ~100M-parameter TTS and a ~60M-parameter ASR), which takes about a minute.
If the Claude session you want to reach runs on a remote host, you can then click on the "⚙" button to scan it. Afterwards, you can select a specific Claude session (or start a fresh Claude "brain") by clicking on 💻.
- Just talk. The ears are always on: a mint mic means listening, red means capturing your sentence. Silero VAD detects speech; stop talking and Moonshine transcribes it.
- Keep talking, it accumulates. Speak again before the reply starts and your words merge into the same prompt instead of resetting it.
- Interrupt it. Talk over it for about half a second and it stops and listens. It measures its own echo through your mic first, so it won't mistake its own voice for yours.
Three sliders in ⚙: hearing (how easily a sound counts as speech, default 50), patience (the pause length that ends your turn, default 800 ms), and interrupt (how long you must talk over it, default 450 ms, 0 disables). There is no reset button: drag back to the defaults, or delete the key (vad_sens, patience_ms, interrupt_ms) from config.json.
To transcribe with something other than Moonshine, add asr_cmd to config.json:
"asr_cmd": "whisper-cli -m ~/models/ggml-base.en.bin -nt -f {wav}"The command receives a 16 kHz mono WAV path and must print the transcript to stdout. Speech detection stays local either way (only transcription is delegated), and Moonshine takes over if the command fails. Remove the line to go back.
Dictation apps like superwhisper replace the keyboard, not this; mute the ears and dictate into the chat bar if that's what you want.
Voice narration can fall behind a fast session. If lines pile up, PocketPal drops the stale ones and jumps to the freshest; set POCKET_WIDGET_STAY_CURRENT=0 to read every line in order instead.
In the settings menu, you can use voice cloning (🎤 button): Record a short audio sample of your voice to use as PocketPal's output. Note that this feature requires the gated pocket-tts model weights: You will ned to accept the terms on Hugging Face and run uvx hf auth login once.
With the claude CLI installed, chat runs through your Pro or Max subscription as headless claude -p calls, no API key. The sessions menu (🧠, top right) picks who answers: a fresh model, a live terminal session, or an ssh session.
No CLI installed? OpenRouter models are listed too. Set $OPENROUTER_API_KEY, or paste a key in ⚙; it is stored plaintext in ~/.cache/pocket_tts_widget/config.json, so keep that file local. Clear the field to remove the stored key.
The sessions menu lists claude CLI sessions already running in your terminals, found via the registry the CLI keeps at ~/.claude/sessions/*.json (either locally or via SSH; you can scan a SSH config in the settings). For each one, the widget checks whether it lives inside a tmux pane it can reach:
- 🖥 full remote (a tmux pane was found, either locally or via SSH): your transcribed sentence is typed straight into the pane with
tmux send-keys -l(a literal, non-shell paste), followed by Enter. The widget then tails that session's own transcript under~/.claude/projects/…and speaks each assistant sentence as it lands, skipping code blocks, tables, and tool records. The session itself can't tell the difference from you typing: it thinks, runs its tools, and edits files with its own model and context. - 👓ride-along (no reachable pane): each turn forks a throwaway
claude --fork-sessioncopy of the session. The fork sees everything typed so far and has real tool access in the project folder, but it is discarded after the turn and your actual terminal is never touched; voice continuity comes from a short recap.
A live session keeps its own model; The sessions menu only chooses which session, or which fresh model, answers.
List ssh hosts in ⚙ (comma-separated, as named in ~/.ssh/config); scan checks them right away. Discovery and driving each run one fixed shell script over ssh (the same registry/tmux/transcript dance as the local case), and the widget reports out loud what it found.
To encourage brevity of the model's answers that will be read by the TTS, one fixed line is appended to every message typed in, e.g. a concrete ban on lists, headers, code blocks, and recaps, rather than a vague "be brief". Edit, disable, or reset it from the ⚙ panel's hint row, or via drive_hint in config.json: a string is a fixed override, "" disables it, a missing key means "use the default".
- A live-session brain turns your always-on mic into a remote shell for that project:
Bash,Edit, andWriteare pre-approved in these headless calls. Only pick this brain when you're comfortable with that, and mute the mic (🎤) or switch to a fresh brain when you step away. Full threat model can be found in SECURITY.md. - Only hosts you list are ever contacted, and
-oBatchMode=yesmeans existing keys/agent only: no password prompts, nothing stored. The model never composes ssh commands; the only variable that crosses the wire is your control-char-scrubbed, shell-quoted message. Host names starting with-are rejected (ssh has no--end-of-options marker). Connections are multiplexed per host, so repeated scans don't re-authenticate.
📜 opens a small window with the full conversation; copy all grabs it as plain text. It lives in memory only; a fresh start or a mascot switch clears it.
Five mascots or "persona" ship built in: Clippy (the default), Pocket, Tama, MoshiVis, and Imp, each with its own personality, color, and default voice.
- Personality: The editor in ⚙ exposes the system prompt behind the current mascot. Rewrite it, save, and it's yours; Reset brings the original back. Any change starts a fresh conversation.
- Voice: Switching mascots resets the voice to that mascot's default (edit
DEFAULT_VOICESinpocketpal/personas.pyto change those); pick a different voice afterward and it sticks until the next mascot switch.
Everything persists in ~/.cache/pocket_tts_widget/: config.json for the ⚙ panel, plus the cloned voice and pasted screenshots. Most settings reset from the panel; For the rest, delete the key from config.json. To reset everything, quit and delete config.json, it will rebuild on then next launch.
One module per concern in pocketpal/: config, text, personas, brains, sessions, audio, engine, api, app, selftest. The UI is ui.html, plus a small second page, history.html, for the history window. The developer guide covers architecture and data flow; SECURITY.md records the threat model (e.g., the always-on mic is an unauthenticated command surface).
uv run python -m pocketpal --selftest # headless end-to-end test: TTS, ASR, barge-in,
# speech filter, live chat turn if the CLI is present
uv run python -m pocketpal --smoke 10 # opens the UI, quits after 10 s| Env var | Effect |
|---|---|
POCKET_WIDGET_EARS_DEBUG=1 |
print mic level vs. threshold once a second |
POCKET_WIDGET_NO_VAD=1 |
disable Silero VAD, fall back to energy detection |
POCKET_WIDGET_STAY_CURRENT=0 |
narrate a driven session in strict order |
POCKET_WIDGET_DEBUG=1 |
open the webview inspector |
POCKET_WIDGET_OPAQUE=1 |
solid window if transparency misbehaves |