Skip to content

feat(voice): release a reply's opening chunk at a clause or word boundary, and drop indented code from speech (#13103, #17953) - #17972

Draft
mrveiss wants to merge 38 commits into
mainfrom
issue-13103-tts-fast-start
Draft

mrveiss wants to merge 38 commits into
mainfrom
issue-13103-tts-fast-start

Conversation

@mrveiss

@mrveiss mrveiss commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Thinking Path

#13103. In a streamed reply, the first audio waited for a terminated sentence of 20+ chars: "long pause, then a wall of speech". The fix releases only the opening chunk on a weaker signal and keeps the standard rule for everything after it.

The issue states the real risk: a too-short opening can render in a noticeably different timbre on some TTS models. So the thresholds are named constants, and this PR stays a draft until the owner's listening test against our TTS worker sets them. Per the issue, if the artifact cannot be avoided at a useful length, this closes as not-worth-doing with the measurement recorded.

#17953's remaining criterion (indented code blocks read aloud) lives in the same file, so it rides here. ~~~ fences already landed in #17954.

Merge-order note: ChatInterface.vue is grandfathered by the upcoming frontend file-size ratchet, and this PR may land after it. So the chunking moved into utils/ttsSentences.ts and ChatInterface.vue shrinks (1527 → 1522). The change is valid in either order; after the ratchet, both anchors are lowered to match.

What Changed

  • utils/ttsSentences.ts:
    • extractOpeningChunk(text) releases the opening at the first clause boundary (, ; : — –) past OPENING_CLAUSE_MIN_CHARS (16), or else at a word boundary once OPENING_WORD_MIN_CHARS (48) is reached. It returns null whenever any sentence has terminated, so it never jumps ahead of a short sentence held for merging. It never cuts inside a markdown link.
    • createTtsStream(alreadyOpened) is per-reply chunking: the opening once, then extractCompleteSentences unchanged. A reply already under way (voice enabled mid-reply, or history) never gets an opening.
    • MIN_TTS_SENTENCE_CHARS (20) moved here from ChatInterface.vue, next to the logic it governs.
    • Indented code blocks (4 spaces or a tab, after a blank line, not list content) are dropped from the spoken copy. Undecided line starts (bare indent, a lone list marker, a partial fence run) are held until the next slice, so a line is never classified twice or mistaken for a blank line.
  • components/chat/ChatInterface.vue: the watcher asks _ttsStream.next(newText). The stream is reset alongside _speechShaper per reply, and as already-opened when the cursor is primed.
  • Tests:
    • utils/__tests__/ttsFastStart.test.ts: the clause/word fallback/too-early/too-short cases, suppression by a held short sentence, no cut inside a link, standard chunks after the opening, no fast start for a reply under way, and exact cursor accounting across 1/3/7/50-char deltas.
    • utils/__tests__/ttsSpeechShaping.test.ts: indented code after a blank line, paragraph continuation kept, nested list content kept, streamed indented blocks, and line starts split across slices.

Refs #13103 (code criteria delivered; the listening-test criterion needs the owner)
Closes #17953

Single-issue rationale: this PR delivers #13103's code criteria together with #17953 (same file). #13103 is linked with Refs only because its last criterion is the owner's listening test, which no code change can satisfy.

Verification

  • Pre-push on 23b0d87662: vue-tsc (autobot-frontend): 0 errors, vitest (autobot-frontend): all relevant tests pass, pytest: all relevant tests pass.
  • Locally, the ttsFastStart, ttsSentences, ttsSpeechShaping, ChatInterface and useVoiceConversation suites give 83/83.
  • code-reviewer: the fast-start logic and cursor accounting hold. One high finding (a slice ending in \n made the next line look like it followed a blank line, so indented prose was dropped) and two mediums (a line cut mid-indent or after a list marker; an opening cut inside a link) are fixed, with regression tests.
  • Accepted as safe misses, not fixed: an opening inside a code block is shaped to nothing and spends the fast start; and "e.g. " or a leading "1. " delays the opening until a standard sentence. Both only lose the early start; neither breaks anything.
  • Not done here: the listening test (Voice: first audio waits for a full terminated sentence — no sub-sentence fast start #13103 AC5). Owner: run voice output against the TTS worker with replies that open on a short clause, then adjust OPENING_CLAUSE_MIN_CHARS / OPENING_WORD_MIN_CHARS and record the chosen values on Voice: first audio waits for a full terminated sentence — no sub-sentence fast start #13103.

Model Used

Claude Opus 5.5 (claude-opus-5-5); review subagent on Sonnet.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Spoken responses can now begin before a full sentence is complete, helping audio playback start sooner on longer replies while avoiding cuts inside Markdown links.
    • Speech output now skips standalone code blocks and waits for streamed formatting to become clear, while retaining relevant prose and indented list content.
    • Subsequent text in a response continues to be spoken at sentence boundaries.

@coderabbitai

coderabbitai Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration
  • Configuration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: b0e20e8e-0502-45ac-880f-03eb20bfcbb9

📥 Commits

Reviewing files that changed from the base of the PR and between 231a607 and 7352b94.


You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough

Walkthrough

The speech shaper now filters fenced and indented code across streamed slices. A new TTS stream supports opening chunks and tracks consumed text. ChatInterface uses the stream to identify speakable spans.

Changes

Streaming text-to-speech

Layer / File(s) Summary
Streamed Markdown code filtering
autobot-frontend/src/utils/ttsSentences.ts, autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts
The speech shaper holds incomplete line starts across streamed slices and filters fenced and eligible indented code. Tests cover indented code, prose continuations and list content.
Opening chunks and stream cursors
autobot-frontend/src/utils/ttsSentences.ts, autobot-frontend/src/utils/__tests__/ttsFastStart.test.ts
The new stream emits terminated sentences first and can emit an opening chunk at an eligible clause or word boundary. Tests cover boundary thresholds, Markdown links, stream behaviour and cursor accounting.
ChatInterface stream integration
autobot-frontend/src/components/chat/ChatInterface.vue
ChatInterface creates and resets a TTS stream for messages. It uses the stream’s spans and consumed length, and primes existing messages in already-consumed mode.

Priority: ⬇️ Low

Severity of issue fixed: Low

Merge Risk: 🔵 Low · up to 7ae62

Code nested inside a list can still be spoken. Fix that filtering gap before merging, or explicitly accept the limited impact.

Pre-merge checks | Passed 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly summarises both main changes: opening-chunk release at clause or word boundaries and removal of indented code from speech.
Linked Issues check Passed #17953 requires removal of ~~~ fences, including split markers without swallowing strikethrough, removal of indented code blocks, preservation of indented prose and nested list content, and tests. `…
Out of Scope Changes check Passed The speech-shaping changes directly implement #17953. The opening-chunk stream changes are explicitly part of the PR's stated #13103 objective, and the stream extraction, cursor handling, integration,…
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 3 files. (1 skipped: 1 …

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR

🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…dary, and drop indented code from speech (#13103, #17953)

#13103: the first audio of a streamed reply waited for a terminated
sentence of 20+ chars. createTtsStream (utils/ttsSentences) releases the
OPENING chunk at the first clause boundary past OPENING_CLAUSE_MIN_CHARS,
or at a word boundary once OPENING_WORD_MIN_CHARS is reached; everything
after keeps the terminated-sentence rule. The fast path never fires when a
sentence has terminated (a short one is held for merging), never for a
reply already under way, and the cursor advances by the exact span.
The thresholds are model-specific and are settled by a listening test
against our TTS worker before merge. ChatInterface delegates to the stream
and its MIN chars constant moves next to the sentence logic (1527 -> 1522).

#17953: indented code blocks are dropped from the spoken copy -- a run of
4-space/tab lines after a blank line that is not list content.
@mrveiss
mrveiss force-pushed the issue-13103-tts-fast-start branch from 23b0d87 to 2007be4 Compare October 4, 2026 20:08
mrveiss and others added 26 commits October 5, 2026 09:28

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: fe91eecc-1591-4013-bb64-7021e4e3a86e
📥 Commits

Reviewing files that changed from the base of the PR and between 1c8f70b and 7ae620f.

📒 Files selected for processing (4)
  • autobot-frontend/src/components/chat/ChatInterface.vue
  • autobot-frontend/src/utils/__tests__/ttsFastStart.test.ts
  • autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts
  • autobot-frontend/src/utils/ttsSentences.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment on lines +151 to +158
recordLine(line: string, eligible: boolean): boolean {
const blank = line.trim() === ''
const indented = INDENTED_RE.test(line)
if (inCode && !blank && !indented) inCode = false
if (!inCode && eligible && prevBlank && indented && !blank && !inList) inCode = true
if (!blank) inList = LIST_ITEM_RE.test(line) || (inList && indented)
prevBlank = blank
return !inCode

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n '110,218p' autobot-frontend/src/utils/ttsSentences.ts
sed -n '101,140p' autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts

Repository: mrveiss/AutoBot-AI

Length of output: 6349


🏁 Script executed:

sed -n '1,115p' autobot-frontend/src/utils/ttsSentences.ts
sed -n '140,170p' autobot-frontend/src/utils/ttsSentences.ts
sed -n '90,145p' autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts
git diff --unified=80 1c8f70b586bfcae61d262023730dcacd8399410d 7ae620f3dcdc11f57340f19451d703cf8a59074f -- autobot-frontend/src/utils/ttsSentences.ts autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts

Repository: mrveiss/AutoBot-AI

Length of output: 27917


🌐 Web query:

CommonMark specification indented code block inside list item blank line 8 spaces

💡 Result:

In CommonMark, **eight spaces before code text can be correct inside a list item**, but the exact threshold depends on the list marker. For `- `, the list content starts after 2 columns, so an indented code block needs 4 more spaces: **6 spaces total**. Eight spaces also works, with the extra two spaces preserved in the code content. ([spec.commonmark.org](https://spec.commonmark.org/spec?utm_source=openai))

A **truly blank line doesn’t need indentation**. If you mean a line containing eight spaces, it counts as blank; inside a code block, blank lines are preserved. ([spec.commonmark.org](https://spec.commonmark.org/spec?utm_source=openai))

```markdown
- item

      code

      more code
```

This puts an indented code block inside the list item. A blank line is needed before the code block when it follows paragraph text. ([spec.commonmark.org](https://spec.commonmark.org/spec?utm_source=openai))

Citations:

- 1: https://spec.commonmark.org/spec?utm_source=openai
- 2: https://spec.commonmark.org/spec?utm_source=openai
- 3: https://spec.commonmark.org/spec?utm_source=openai

Filter indented code blocks inside list items.

After - item\n\n, inList remains true. The eight-space secret() line therefore bypasses the !inList check and reaches speech output. CommonMark treats this as an indented code block inside the list item, not list prose. Track the list item's content indentation so the existing four-space nested-list case remains spoken.

Suggested fix
diff --git a/autobot-frontend/src/utils/ttsSentences.ts b/autobot-frontend/src/utils/ttsSentences.ts
@@
 const INDENTED_RE = /^(?: {4}| {0,3}\t)/
-const LIST_ITEM_RE = /^[ \t]*(?:[-*+]|\d+[.)])[ \t]/
+const LIST_ITEM_RE = /^[ \t]*(?:[-*+]|\d+[.)])[ \t]+/
+const leadingIndent = (line: string): number => line.match(/^[ \t]*/)?.[0].length ?? 0
 
 function createIndentedCodeTracker() {
   let prevBlank = true
   let inList = false
+  let listContentIndent: number | null = null
   let inCode = false
   return {
@@
       const blank = line.trim() === ''
       const indented = INDENTED_RE.test(line)
+      const indent = leadingIndent(line)
+      const listItem = LIST_ITEM_RE.test(line)
       if (inCode && !blank && !indented) inCode = false
-      if (!inCode && eligible && prevBlank && indented && !blank && !inList) inCode = true
-      if (!blank) inList = LIST_ITEM_RE.test(line) || (inList && indented)
+      if (
+        !inCode &&
+        eligible &&
+        prevBlank &&
+        indented &&
+        !blank &&
+        (!inList || listContentIndent === null || indent >= listContentIndent + 4)
+      ) inCode = true
+      if (!blank) {
+        if (listItem) listContentIndent = line.match(LIST_ITEM_RE)![0].length
+        inList =
+          listItem ||
+          (inList && indented && listContentIndent !== null && indent < listContentIndent + 4)
+      }
       prevBlank = blank
       return !inCode
@@
       prevBlank = true
       inList = false
+      listContentIndent = null
       inCode = false
diff --git a/autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts b/autobot-frontend/src/utils/__tests__/ttsSpeechShaping.test.ts
@@
   it('keeps indented list content, nested items included (#17953)', () => {
     expect(shape('- item\n\n    - nested item\n    second para')).toBe('item nested item second para')
   })
 
+  it('drops indented code nested inside a list item (#17953)', () => {
+    expect(shape('- item\n\n        secret()\n\nAfter.')).toBe('item After.')
+  })
+

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

voice: the speech shaper drops ``` fences but still reads ~~~ fences and indented code blocks aloud

1 participant