Skip to content

feat(asr): add ElevenLabs Scribe v2 provider#807

Merged
appergb merged 3 commits into
betafrom
fix/issue-795-elevenlabs-scribe
Jul 14, 2026
Merged

feat(asr): add ElevenLabs Scribe v2 provider#807
appergb merged 3 commits into
betafrom
fix/issue-795-elevenlabs-scribe

Conversation

@appergb

@appergb appergb commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

User description

Summary

Replaces #797 with a clean branch based on the current beta tip.

Fixes #795.

Adds ElevenLabs Scribe v2 as a batch ASR provider and wires it through dictation, QA, retranscription, cancellation, provider validation, provider-scoped credentials, settings, and localized provider labels.

Review finding fixed

ElevenLabs defaults timestamps_granularity to word, which adds a potentially large words array even though OpenLess only consumes the top-level transcript text. The client now explicitly sends timestamps_granularity=none, keeping long-recording responses text-only and within the existing defensive 1 MiB response cap. A boundary test proves that exactly 1 MiB is accepted and the next byte is rejected.

Official API reference: https://elevenlabs.io/docs/api-reference/speech-to-text/convert

Security and compatibility

  • Preserves HTTPS-only remote endpoints, with loopback HTTP only for local development.
  • Preserves DNS resolution pinning and blocks sensitive/private endpoint targets.
  • Does not follow redirects with xi-api-key or audio.
  • Preserves bounded connect/request timeouts, cancellation, sanitized error bodies, and provider-scoped credentials.
  • Defaults to the current scribe_v2 model.
  • Includes only the existing small provider settings additions; no broad UI redesign.
  • Do not merge both this PR and feat(asr): 新增 ElevenLabs Scribe 语音转录供应商 #797.

Verification

  • TDD red: the multipart regression test failed on the imported feat(asr): 新增 ElevenLabs Scribe 语音转录供应商 #797 implementation because timestamps_granularity was absent.
  • TDD green: cargo test --locked --manifest-path src-tauri/Cargo.toml asr::elevenlabs::tests — 12 passed, 0 failed.
  • cargo test --locked --manifest-path src-tauri/Cargo.toml — 625 passed, 0 failed.
  • npm ci --no-audit --no-fund — completed.
  • npm run build — completed; only the pre-existing Android IPC manual-chunk circular warning remains.
  • Contract checks passed: macOS capsule Spaces, hotkey injection, side modifiers, Windows startup lifecycle, Android updater pubkey.
  • npm audit --json — 0 vulnerabilities.
  • git diff --check origin/beta...HEAD — clean.
  • qwen-asr submodule initialized at b00b789b17051aea61e9717458171100662318a4.

GitNexus classifies the complete provider integration as critical blast radius because it touches 34 execution flows across dictation, QA, cancellation, and provider dispatch. The focused reviewer-finding fix itself is low risk (one file, eight symbols, zero mapped flows). This PR requires independent agent review before merge.


PR Type

Enhancement, Tests, Documentation


Description

  • Add ElevenLabs Scribe batch ASR client

  • Wire through dictation, QA, retranscribe, and cancel

  • Extract SSRF validation to shared module

  • Add per-provider credential isolation and frontend preset

  • Update i18n for 5 languages and privacy descriptions


Diagram Walkthrough

flowchart LR
    AudioBuffer["Audio Buffer (16kHz PCM)"] --> EncodeWav["Encode WAV"]
    EncodeWav --> MultipartForm["Multipart POST /v1/speech-to-text"]
    MultipartForm --> SecCheck["SSRF Check & Redirect Block"]
    SecCheck --> ElevenLabsAPI["ElevenLabs API"]
    ElevenLabsAPI --> ParseResponse["Parse JSON Response"]
    ParseResponse --> Transcript["Transcript Text"]
    Transcript --> App["Application"]
Loading

File Walkthrough

Relevant files
Enhancement
15 files
elevenlabs.rs
New ElevenLabs Scribe ASR client                                                 
+550/-0 
mod.rs
Export ElevenLabs module                                                                 
+2/-0     
credentials.rs
Add per-provider credential read/write                                     
+32/-4   
providers.rs
Add ElevenLabs provider validation                                             
+53/-11 
coordinator.rs
Integrate ElevenLabs ASR into coordinator                               
+43/-84 
asr_wiring.rs
Add ElevenLabs provider detection                                               
+11/-0   
dictation.rs
Handle ElevenLabs dictation sessions                                         
+36/-0   
qa_session.rs
Handle ElevenLabs QA transcription with cancel                     
+191/-5 
resources.rs
Add ElevenLabs cancel support                                                       
+1/-0     
endpoint_security.rs
Extract SSRF validation module                                                     
+109/-0 
lib.rs
Register endpoint_security module                                               
+1/-0     
credentials.rs
Add per-provider credential methods                                           
+18/-0   
asr-credentials.ts
Update credential IPC for provider scope                                 
+4/-4     
ProvidersSection.tsx
Add ElevenLabs preset and per-provider fields                       
+31/-17 
shared.tsx
Add ElevenLabs ASR preset definition                                         
+1/-0     
Documentation
5 files
en.ts
Add English i18n for ElevenLabs                                                   
+5/-3     
ja.ts
Add Japanese i18n for ElevenLabs                                                 
+5/-3     
ko.ts
Add Korean i18n for ElevenLabs                                                     
+5/-3     
zh-CN.ts
Add Simplified Chinese i18n for ElevenLabs                             
+5/-3     
zh-TW.ts
Add Traditional Chinese i18n for ElevenLabs                           
+5/-3     

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit 76603c5)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

795 - Partially compliant

Compliant requirements:

  • 新增独立 ElevenLabs ASR provider,使用官方 xi-api-key + multipart Speech-to-Text 协议。
  • 默认 endpoint 为 https://api.elevenlabs.io/v1,默认模型为 scribe_v2,允许用户自定义 endpoint/model。
  • 覆盖普通听写、QA、重转录和取消路径,不影响现有 provider。
  • 设置页提供 ElevenLabs 预设,并补齐现有五种语言的供应商名称。
  • API key 仅通过现有凭据存储读取,不进入日志、错误证据或仓库。
  • Rust 单测、前端构建、cargo check 和上游多平台 CI 通过。
  • 至少完成一次真实 macOS 全链路语音验证,公开结果只保留脱敏摘要。

Non-compliant requirements:

(empty)

Requires further human verification:

(empty)

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected

@appergb appergb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES

我独立复核了 issue #795 对应的完整 20 文件 functional diff。当前有 1 个 P2 阻塞项:

[P2] QA 浮窗中的当前听写在 ElevenLabs 转写阶段取消,会被展示为“识别失败”

位置:openless-all/app/src-tauri/src/coordinator/qa_session.rs:288-300,以及同文件 138-145。

wait_for_overlay_dictation_cancel 返回后,ElevenLabs 分支先调用 asr.cancel(),随后返回 Err("dictation cancelled")。上层 take_current_dictation_transcript_for_qa 对所有 Err 都调用 finish_qa_with_error("识别失败: ...") 并直接返回,因此 148-156 行原本用于静默处理取消的 cancelled 检查永远不会执行。结果是请求虽然停止了,但用户主动取消会得到错误胶囊,违背 #795 的取消行为要求。

请把“用户取消”与“转写失败”分开表达,例如使用结构化结果;或者在上层 Err 分支先检查 cancelled / session_id 是否已变化,恢复状态并返回 Ok(None),不要调用 finish_qa_with_error。请补一个回归测试:在 QA 浮窗的 ElevenLabs 当前听写转写阶段触发取消,断言最终为 idle 且不发 Error/识别失败事件。

非阻塞:

[P3] 新增的 persistence/credentials.rs:958 set_for_asr_provider 签名不符合当前 rustfmt 输出。仓库 beta 本身已有大量格式债务,但 cargo fmt --check 对 PR 新增这一处也给出 diff,建议随修复一起格式化该函数。

其余独立核验:

  • ElevenLabs 官方 API 文档确认 POST /v1/speech-to-text、xi-api-key、multipart file/model_id、scribe_v2 及 timestamps_granularity=none;本实现匹配。
  • 检查了 HTTPS/loopback 限制、DNS 固定、禁止重定向、动态超时、1 MiB 流式响应上限、错误体脱敏与 API key 不落日志。
  • 检查了 provider-scoped 凭据读写和 active provider 路由。CredentialsVault::get 会按 root.active.asr 取当前 provider,而切换时先更新 active;未复现机器人评论所述的凭据错配。
  • 检查了普通听写、QA、重转写和资源取消路由,以及 5 个 locale 和设置页变更;未见大范围 UI 重构。
  • CI run 29311411804 在当前 head 3f87cfd 上全绿:macOS 625/625、Linux 611/611、Windows Rust-only 123/123,Android cargo check 通过;12 个 ElevenLabs 单测在 macOS/Linux 通过。
  • git diff --check 通过,差异中未发现硬编码密钥。cargo fmt --check 整体仍失败,主要是 beta 既有格式债务,另含上面的新增 P3。

当前 gh 账号 appergb 与 PR 作者相同,GitHub 不允许正式 APPROVE/REQUEST_CHANGES,因此此处按要求提交 COMMENTED review;审阅结论仍是 REQUEST_CHANGES,修复 P2 后再批准。

@H-Chris233

Copy link
Copy Markdown
Collaborator

快修p2
不然我把你全覆盖了

@appergb
appergb force-pushed the fix/issue-795-elevenlabs-scribe branch from 3f87cfd to 76603c5 Compare July 14, 2026 07:16
@github-actions

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 76603c5

@appergb appergb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

复审 head 76603c5 后,没有发现新的 P1/P2/P3 阻塞项。此前 review #4691507297 的两个问题均已解决:

  • P2 取消错误路径已修复:transcribe_overlay_dictation_asr 现在通过 OverlayDictationTranscribeOutcome::Cancelled 显式区分用户取消和转写失败;take_current_dictation_transcript_for_qa 对取消路径恢复会话并调用 finish_qa_idle_silently,返回 Ok(None),不再进入 finish_qa_with_error。
  • 新增 overlay_elevenlabs_cancel_finishes_idle_without_error_capsule 回归测试,实际启动 ElevenLabs 请求后触发 cancel_session,并断言主会话与 QA 都为 Idle、QA 取消标志复位、最后胶囊状态为 Idle。旧实现会停在 Error,因此该测试能捕获原回归。
  • P3 rustfmt 已修复:set_for_asr_provider 现在与 rustfmt 输出一致。cargo fmt --check 仍会报告 qa_session.rs:60 和 credentials.rs 的既有 beta 格式债务,但不再报告此前新增函数。
  • ElevenLabs 响应测试 helper 改写保持生产边界:read_response_limited 对每个 reqwest bytes_stream chunk 都调用生产函数 append_response_chunk;该函数以累计长度限制 MAX_RESPONSE_BYTES。测试验证恰好 1 MiB 可接受,随后再加 1 byte 会拒绝,等价覆盖生产累计边界,不是测试专用旁路。

验证结果:

  • 对比旧 head 3f87cfd 与新 head,实际修复仅涉及 qa_session.rs、persistence/credentials.rs 和 elevenlabs.rs;完整 PR 仍为 20 文件。
  • git diff --check origin/beta...HEAD 通过,独立 worktree 无修改。
  • 当前 PR mergeable。分支比 beta 少安全依赖提交 66d0847,但 GitHub CI 使用的 merge ref b307c4e 的两个父提交正是 beta 66d0847 与 head 76603c5,因此已验证实际合并结果。
  • CI run 29314019430 全绿:Android cargo check、Linux、macOS、Windows。Linux 612/612、macOS 626/626、Windows Rust-only 123/123;两个重点测试 response_limit_accepts_one_megabyte_and_rejects_the_next_byte 与 overlay_elevenlabs_cancel_finishes_idle_without_error_capsule 在 Linux/macOS 均通过。
  • PR-Agent 检查也通过。

当前 gh 账号 appergb 与 PR 作者相同,GitHub 不允许正式 APPROVE 自己的 PR,所以按要求提交 COMMENTED review;逻辑结论为 APPROVE,代码层面可合并,仍需其他账号完成仓库要求的正式批准。

@appergb
appergb merged commit 145a0ea into beta Jul 14, 2026
5 checks passed
@appergb
appergb deleted the fix/issue-795-elevenlabs-scribe branch July 14, 2026 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[asr] 新增 ElevenLabs Scribe 语音转录供应商

2 participants