feat(expressive): allow dynamic updates via Agent and updateOptions - #2297
feat(expressive): allow dynamic updates via Agent and updateOptions#2297rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: e72bfc5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 37 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| export interface NonverbalOptions { | ||
| laughing?: boolean; | ||
| breathing?: boolean; | ||
| sighing?: boolean; | ||
| crying?: boolean; | ||
| vocalizing?: boolean; | ||
| mouthSounds?: boolean; | ||
| reflexSounds?: boolean; | ||
| } | ||
|
|
||
| export interface SpeechSteeringOptions { | ||
| disfluencies?: boolean; | ||
| nonverbalSounds?: boolean | NonverbalOptions; | ||
| pace?: 'slow' | 'normal' | 'fast'; | ||
| } |
There was a problem hiding this comment.
🟡 New public configuration types ship without documentation
The newly exported expressive configuration types and their fields are added without TypeDoc comments (NonverbalOptions / SpeechSteeringOptions at agents/src/voice/agent_session.ts:126-140), so the generated API docs will show these public options with no explanation.
Impact: Users browsing the published API reference see undocumented settings and cannot tell what they do.
Repository rule requiring documentation of new public types
CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented." NonverbalOptions and SpeechSteeringOptions are exported publicly from agents/src/voice/index.ts:25-26 but carry no doc comments, unlike the sibling ExpressiveOptions which does.
| export interface NonverbalOptions { | |
| laughing?: boolean; | |
| breathing?: boolean; | |
| sighing?: boolean; | |
| crying?: boolean; | |
| vocalizing?: boolean; | |
| mouthSounds?: boolean; | |
| reflexSounds?: boolean; | |
| } | |
| export interface SpeechSteeringOptions { | |
| disfluencies?: boolean; | |
| nonverbalSounds?: boolean | NonverbalOptions; | |
| pace?: 'slow' | 'normal' | 'fast'; | |
| } | |
| /** Toggles for individual nonverbal sounds an expressive TTS may produce. */ | |
| export interface NonverbalOptions { | |
| laughing?: boolean; | |
| breathing?: boolean; | |
| sighing?: boolean; | |
| crying?: boolean; | |
| vocalizing?: boolean; | |
| mouthSounds?: boolean; | |
| reflexSounds?: boolean; | |
| } | |
| /** Controls how speech is steered (disfluencies, nonverbal sounds, pace). */ | |
| export interface SpeechSteeringOptions { | |
| disfluencies?: boolean; | |
| nonverbalSounds?: boolean | NonverbalOptions; | |
| pace?: 'slow' | 'normal' | 'fast'; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6886 to expose expressive settings on Agent and both runtime update APIs. Agent-level values override the session setting, and live agent updates apply expressive only after model-swap validation succeeds.
Infrastructure gap
The JS SDK does not yet contain the pre-existing expressive rendering pipeline that Python #6886 builds on: provider markup dialects/conversion, XML-aware tokenization, prompt injection, and transcript/history markup stripping. This PR ports the dynamic configuration and effective resolution from #6886, but does not enable markup rendering itself; adding that roughly 4,000-line prerequisite feature is not feasible as part of this focused dynamic-update port and should be handled in a dedicated parity PR.
Source diff coverage
Source diff coverage
livekit-agents/livekit/agents/voice/agent.py: ported toagents/src/voice/agent.ts. Adds the constructor option, getter, and dynamic update behavior, including applying expressive only after successful live model validation.livekit-agents/livekit/agents/voice/agent_activity.py: adapted toagents/src/voice/agent_activity.ts. Adds agent-over-session resolution and inference-provider capability gating in the target architecture. The target lacks the upstream pre-existing rendering infrastructure described above.livekit-agents/livekit/agents/voice/agent_session.py: ported toagents/src/voice/agent_session.ts,agents/src/voice/turn_config/utils.ts, andagents/src/voice/index.ts. Adds public option types, the default-off session state, runtime updates, and exports.tests/test_expressive_toggle.py: adapted toagents/src/voice/expressive_toggle.test.ts. Ports exactly the three tests added by #6886: session updates, agent updates, and agent-over-session precedence.Validation
pnpm test -- agents/src/voice/expressive_toggle.test.ts agents/src/voice/agent_update_options.test.ts agents/src/voice/agent_session_endpointing.test.ts(34 passed)pnpm test -- agents --exclude agents/src/voice/amd.test.ts(1,261 passed, 5 skipped)pnpm buildpnpm lint(passes with existing warnings)pnpm format:checkpnpm --filter @livekit/agents typecheckThe complete
agentssuite was run. Its unrelatedagents/src/voice/amd.test.tsbaseline has 11 reproducible failures when run alone (classifier fallbacks/timeouts); the other 1,261 tests pass. API Extractor is also blocked by its existing lack of support for the generatedexport * asdeclaration.cue-cliis installed, but runtime dispatch validation could not run because this environment has no LiveKit connection credentials.Ported from livekit/agents#6886
Original PR description
expose expressive= on AgentSession.update_options and add it to Agent (constructor + update_options). The agent value overrides the session's, matching how the agent's llm/tts override the session models; resolution stays per-turn so changes apply on the next reply.
https://community.livekit.io/t/making-voice-agents-sound-human-with-expressive-mode/1854/3