feat: add wake phrase voice assistant preview

[calebeden]

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers can help update the branch when needed.

What Problem This Solves

Windows users do not have an always-on, wake-phrase voice experience comparable to the macOS app. Existing speech input, gateway chat, and speech output work independently, but users must manually start each interaction.

Why This Change Was Made

This adds a default-off Assistant preview that reuses the existing local Whisper STT, energy VAD, gateway chat path, and configured TTS provider. V0 deliberately supports one flow: say the wake phrase and request in one utterance, receive one spoken reply in the currently selected native Chat session, then return to listening.

Each Assistant request owns exactly one gateway-accepted run. Response claiming, cancellation, terminal handling, and late-response suppression are scoped to that run so unrelated activity in the same session is not spoken or aborted. Turning the feature off disposes its coordinator, subscriptions, and microphone capture machinery.

V0 adds no wake-word model, protocol command, persistent conversation mode, barge-in, or overlay.

User Impact

Users can configure a custom one-to-three-word wake phrase on Voice & Audio after configuring speech input and Companion Voice. When enabled, a matching utterance sends only the request after the wake phrase, capitalizes its first character, routes it through the visible selected Chat session, speaks the matching reply once, and resumes listening.

While Assistant owns the microphone for wake listening, native Chat visibly explains why single-message voice input is unavailable. Switching sessions does not restart wake capture. Turning Assistant off releases capture and restores normal voice transcription.

Evidence

• Full repository build passed on Windows ARM64 with .NET SDK 10.0.301.
• Shared tests: 3,399 passed, 32 skipped.
• Tray tests: 2,107 passed, 0 skipped.
• Focused voice and run-correlation tests: 34 passed.
• The contributor manually retested the complete user-visible wake-to-selected-session-to-spoken-reply flow on current head 31c90691 and confirmed that it matches the existing recording.
• Automated coverage includes wake matching, settings persistence, readiness, exclusive microphone ownership, selected-session routing, accepted-run binding, pre-binding response buffering, cross-run isolation, exact cancellation, UI Stop ordering, terminal/final event reordering, deduplication, timeout, TTS recovery, Mode Off disposal, localization, and accessibility feedback.
• Current-head CI passed the revocation-recovery, setup-connect, and network-recovery E2E jobs. The aggregate test job is currently red because the unrelated SystemRun_Where_ReturnsExpectedOutput Tray integration test returned exit code -1 instead of 0.

Change Type

• Bug fix
• Feature
• Refactor
• Docs or instructions
• Tests or validation
• Security hardening
• Chore or infrastructure

Scope

• Tray or WinUI UX
• Windows node capability
• Local MCP or winnode
• Gateway, connection, or pairing
• Setup or onboarding
• Permissions, privacy, or security
• Tests, CI, or docs

Validation

• ./build.ps1: passed
• dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore: 3,399 passed, 32 skipped
• dotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj --no-restore: 2,107 passed, 0 skipped
• Focused voice and run-correlation selection from OpenClaw.Tray.Tests: 34 passed
• GitHub Actions e2etests (revocation-recovery, ...): passed on 31c90691
• GitHub Actions e2etests (setup-connect, ...): passed on 31c90691
• GitHub Actions e2etests (network-recovery, ...): passed on 31c90691
• GitHub Actions aggregate test: failed in unrelated McpHttpServerIntegrationTests.SystemRun_Where_ReturnsExpectedOutput; Shared, Tray, Connection, and WinNode CLI test steps passed before that failure

Real Behavior Proof

• Environment tested: Windows 11 ARM64, .NET SDK 10.0.301, Windows SDK 10.0.26100.0
• PR head tested: 31c906917f00b369bde04b928e29d721be7dfd02
• Exact current-head steps: Enabled Assistant with configured STT and TTS, spoke the configured wake phrase and request, switched native Chat sessions, and confirmed that the request routed to the visible selected session with one matching TTS playback. Capture resumed normally, and switching sessions did not cause a microphone restart loop.
• Existing recording of the same wake-to-request-to-spoken-reply user experience:

OpenClawAssistantV0.mp4

• Recording applicability: The recording was captured on an earlier commit, but the complete user-visible flow was manually repeated on 31c90691 and produced the same behavior. Changes after the recording harden internal run correlation, cancellation, event ordering, and disposal without changing the demonstrated interaction.
• Current-head automated proof: exact-run regressions cover unrelated same-session finals, final/terminal ordering before and after accepted-run binding, cancellation while another run is active, UI Stop before a late final, readiness changes during the owned run, and Mode Off disposal.
• Observed result: Wake requests route through the selected Chat session, only the owned response is eligible for speech, and capture resumes after the turn.
• Screenshot or artifact links verified? (Yes/No/N/A): Yes, the recording resolves as an MP4 and remains representative of the manually retested current-head UX
• Not verified or blocked: No user-visible flow is blocked or unverified on current head. Concurrent-run isolation, exact cancellation isolation, and Mode Off disposal are internal invariants covered by current-head automated tests rather than separately visible in the recording.

Security Impact

• New permissions or capabilities? (Yes/No): No
• Secrets or tokens handling changed? (Yes/No): No
• New or changed network calls? (Yes/No): No, requests use the existing configured gateway chat path
• Command or tool execution surface changed? (Yes/No): No
• Data access scope changed? (Yes/No): Yes
• If any answer is Yes, explain the risk and mitigation: When explicitly enabled, the microphone remains active and local Whisper transcribes silence-bounded speech bursts to detect the configured phrase. The feature is off by default, requires existing STT and TTS capabilities, does not log audio or transcript content, and sends only the extracted request through the existing configured gateway path.

Compatibility and Migration

• Backward compatible? (Yes/No): Yes
• Config or environment changes? (Yes/No): Yes, two optional Assistant settings are persisted with safe defaults
• Migration needed? (Yes/No): No
• If yes, list the exact upgrade steps: N/A

Review Conversations

• I replied to or resolved every bot review conversation addressed by this PR.
• I left unresolved only conversations that still need maintainer judgment.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 9, 2026, 6:31 PM ET / 22:31 UTC.

ClawSweeper review

What this changes

Adds a default-off Windows wake-phrase assistant that transcribes one request, routes it through the selected native chat session, and speaks the correlated reply.

Regression provenance

Possible regression — probable (reviewed change; known regression link). No predecessor PR is attributed.

Merge readiness

⛔ Blocked until stronger real behavior proof is added - 10 items remain

Keep this PR open for maintainer review. It has a concrete wake-capture recovery defect and needs current-head Windows proof after resolving its dirty merge state; the linked microphone freeze fix is now on main but was not in the PR head.

Priority: P1
Reviewed head: 31c906917f00b369bde04b928e29d721be7dfd02
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The feature has substantial tests and an earlier recording, but a P1 recovery flaw and absent current-head visible proof keep it below merge readiness.
Proof confidence	🦪 silver shellfish (2/6)	Needs stronger real behavior proof before merge: The linked recording is explicitly from an earlier revision, while current-head behavior is only asserted; add redacted current-head Windows evidence for normal wake flow and watchdog recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs stronger real behavior proof before merge: The linked recording is explicitly from an earlier revision, while current-head behavior is only asserted; add redacted current-head Windows evidence for normal wake flow and watchdog recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	6 items	Current-main no-data safeguard: Current main arms a first-audio watchdog, transitions to Error, releases the capture, and emits an actionable diagnostic when WASAPI supplies no data. Watchdog provenance: The silent-WASAPI safeguard was introduced by the current-main commit after this PR head. Main-only release check: No release tag contains the watchdog commit; local containment shows it is on main.
Findings	1 actionable finding	[P1] Propagate the no-data watchdog to wake recovery
Security	None	None.

How this fits together

The tray combines local Whisper microphone capture, native gateway chat, and configured text-to-speech. The assistant owns capture during wake listening, then sends one tracked chat turn and returns to listening after its matching reply.

flowchart LR
A[Microphone] --> B[Local speech capture]
B --> C[Wake phrase gate]
C --> D[Assistant coordinator]
D --> E[Selected chat session]
E --> F[Correlated final reply]
F --> G[Configured speech output]

Loading

Decision needed

Question	Recommendation
Should the Windows tray ship this broad always-on microphone preview once its capture recovery defect and merge conflicts are resolved?	Sponsor the narrowed preview: Keep the one-shot, default-off scope and require the recovery fix, rebase, and current-head Windows proof before merge.

Why: The technical repair is clear, but accepting a new persistent microphone interaction and its preview scope is a product decision.

Before merge

• Add real behavior proof - Needs stronger real behavior proof before merge: The linked recording is explicitly from an earlier revision, while current-head behavior is only asserted; add redacted current-head Windows evidence for normal wake flow and watchdog recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Propagate the no-data watchdog to wake recovery (P1) - Late finding: this adapter is unchanged since the prior reviewed head. After rebasing, current main can asynchronously move AudioPipeline to Error after five seconds with no data, but this new adapter forwards neither that failure nor a state transition to the coordinator. The wake lease remains held until VoiceService.StopAsync, leaving the coordinator in wake listening and normal chat voice input unavailable. Surface the failure, release or reconcile the assistant state, and add a no-data recovery regression.
• Resolve merge risk (P1) - After rebasing onto current main, a no-data microphone can move the pipeline to Error while the assistant remains logically in wake listening and keeps normal chat voice input unavailable.
• Resolve merge risk (P1) - The PR is dirty against current main, so its declared validation and earlier recording do not prove the actual merge result.
• Resolve merge risk (P1) - The default-off feature continuously owns microphone capture when enabled, so the failure and recovery path needs current-head Windows evidence.
• Complete next step (P2) - Maintainer sponsorship is needed for the preview direction; if sponsored, require the contributor to rebase, repair watchdog recovery, and provide current-head Windows proof.
• Improve patch quality - Rebase and resolve conflicts against current main, preserving the silent-capture watchdog.
• Improve patch quality - Add recovery handling and regression coverage for the watchdog Error transition.
• Improve patch quality - Post redacted current-head Windows proof for normal wake flow and no-data capture recovery.

Findings

• [P1] Propagate the no-data watchdog to wake recovery — src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs:20-23

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Change surface	54 files, +4,374/-90 lines	The feature crosses capture, chat, settings, localization, documentation, and tests, so the rebased merge result needs focused review.

Merge-risk options

Maintainer options:

1. Recover cleanly from silent capture (recommended)
   Rebase onto current main, propagate the first-audio watchdog failure to the assistant coordinator, release its capture lease, and prove recovery on Windows.
2. Pause the preview
   Do not merge the feature if maintainers do not want to own an always-on microphone preview at this stage.

Technical review

Best possible solution:

Land only a rebased one-shot preview that forwards capture failure into the coordinator, releases assistant microphone ownership, restores chat voice input, and has current-head Windows evidence for both normal wake flow and no-data recovery.

Do we have a high-confidence way to reproduce the issue?

Yes, source provides a high-confidence path: enable wake listening on a device that produces no initial WASAPI data, wait for the five-second watchdog, and observe whether assistant state and the microphone lease recover.

Is this the best way to solve the issue?

No. Rebase alone retains the watchdog but does not make the new assistant adapter report that asynchronous failure to its coordinator; the feature needs an explicit recovery transition and proof.

Full review comments:

• [P1] Propagate the no-data watchdog to wake recovery — src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs:20-23
  Late finding: this adapter is unchanged since the prior reviewed head. After rebasing, current main can asynchronously move AudioPipeline to Error after five seconds with no data, but this new adapter forwards neither that failure nor a state transition to the coordinator. The wake lease remains held until VoiceService.StopAsync, leaving the coordinator in wake listening and normal chat voice input unavailable. Surface the failure, release or reconcile the assistant state, and add a no-data recovery regression.
  Confidence: 0.88
  Late finding: first raised on code an earlier review cycle already covered.

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 582a44713860.

Labels

Label changes:

• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
• remove rating: 🦐 gold shrimp: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.

Label justifications:

• P1: A wake-listening failure can leave the tray's voice interaction unavailable after microphone capture fails.
• merge-risk: 🚨 availability: This PR adds persistent microphone ownership and currently lacks a recovery transition for the main watchdog error.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The linked recording is explicitly from an earlier revision, while current-head behavior is only asserted; add redacted current-head Windows evidence for normal wake flow and watchdog recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• proof: 🎥 video: Contributor real behavior proof includes video or recording evidence. The linked recording is explicitly from an earlier revision, while current-head behavior is only asserted; add redacted current-head Windows evidence for normal wake flow and watchdog recovery before merge.

Evidence

What I checked:

• Current-main no-data safeguard: Current main arms a first-audio watchdog, transitions to Error, releases the capture, and emits an actionable diagnostic when WASAPI supplies no data. (src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs:764, 582a44713860)
• Watchdog provenance: The silent-WASAPI safeguard was introduced by the current-main commit after this PR head. (src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs:807, 20d29e1132c3)
• Main-only release check: No release tag contains the watchdog commit; local containment shows it is on main. (20d29e1132c3)
• Wake adapter does not expose capture failure: The supplied PR diff shows the new adapter forwards completed utterances and lease availability but no pipeline-error signal to the coordinator. (src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs:20, 31c906917f00)
• Previous-review continuity: The adapter file is unchanged from the previously reviewed head, so the recovery concern remains applicable to this exact revision. (src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs:20, 31c906917f00)
• Proof is not current-head visible evidence: The PR body identifies its recording as from an earlier commit and supplies only an assertion of manual repetition on the current head; the recording could not be retrieved in this review because GitHub DNS resolution failed. (31c906917f00)

Likely related people:

• Scott Hanselman: Authored the current-main silent-WASAPI recovery fix and gave the prior run-correlation review. (role: recent capture-safety contributor and prior reviewer; confidence: high; commits: 20d29e1132c3; files: src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs)
• Régis Brid: Introduced the Whisper STT and Piper TTS service area that owns the microphone pipeline. (role: original audio-service contributor; confidence: medium; commits: b0ba9affa25d; files: src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs, src/OpenClaw.Tray.WinUI/Services/VoiceService.cs)
• Christine Yan: Has recent merged history on speech-input behavior and its tests. (role: recent adjacent speech-input contributor; confidence: medium; commits: 99efc50cbc22; files: src/OpenClaw.Tray.WinUI/Services/VoiceService.cs, tests/OpenClaw.Tray.Tests/SpeechInputContractTests.cs)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (45 earlier review cycles; latest 8 shown)

• reviewed 2026-08-07T16:01:13.877Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Guard wake listening against zero-data capture
• reviewed 2026-08-07T18:41:41.902Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Guard wake listening against zero-data capture
• reviewed 2026-08-07T21:25:25.824Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Guard wake listening against zero-data capture
• reviewed 2026-08-08T14:24:37.371Z sha 31c9069 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-08T15:39:57.567Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase wake capture onto the silent-capture watchdog
• reviewed 2026-08-09T00:06:24.377Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase capture lifecycle onto the silent-capture watchdog
• reviewed 2026-08-09T03:16:26.018Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase wake capture onto the silent-capture watchdog
• reviewed 2026-08-09T14:51:40.695Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase wake capture onto the no-data watchdog

[shanselman]

Adversarial maintainer review found blocking run-correlation defects, so this is not ready to land yet:

• The assistant can claim and speak an unrelated final message from the same thread because correlation is thread-scoped rather than gateway-run-scoped.
• Timeout, TTS failure, disable, or recovery can call the thread-scoped abort path and terminate another active run, or queue an abort against the next run.
• Readiness changes can cancel the assistant's own in-flight turn, and the UI Stop path is not coordinated before a late response is claimed/spoken.
• Turning the feature off reconciles to Off but leaves the coordinator, subscriptions, and capture machinery alive.

Please bind the turn to the gateway run id, make cancellation no-op unless that exact run is still active, move abort suppression ahead of response claiming, and dispose the coordinator when disabled. The current revocation-recovery check is also red. Current merge confidence is 45%.

[calebeden]

Caleb Eden + GitHub Copilot pair-programming follow-up

@shanselman Thanks for the detailed adversarial review. I addressed each of the run-ownership and lifecycle concerns:

• Each Assistant turn is now bound to the authoritative gateway run ID returned by chat.send. Final-response claiming requires the matching session, accepted run, message identity, and sequence boundary, including responses that arrive before run binding completes.
• Assistant cancellation targets only that accepted run. It becomes a no-op if another run is active or the owned run is already terminal, and it does not queue a deferred thread abort.
• Readiness changes preserve the Assistant's own in-flight run. UI Stop invalidates matching Assistant ownership before late responses can be claimed, and cancelled-run suppression is checked before response claiming.
• Switching the mode Off now disposes the coordinator, chat adapter subscriptions, and wake-input capture machinery.

The revocation-recovery E2E check now passes on the current head. The aggregate workflow is still red because of an unrelated Tray integration failure in SystemRun_Where_ReturnsExpectedOutput, so I am not claiming the entire CI workflow is green.

[calebeden]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/30962126123
• Updated: 2026-08-05T00:05:05.744Z

[calebeden]

Note that this should not be merged until microphone bugs are resolved. #1097 describes a UI freeze when attempting to capture audio input. Merging this with that bug present could lead to a bricked installation where a user enables voice assistant, it starts persistently listening, and then the UI freezes and the user is unable to turn off the voice assistant.