min_endpointing_delay behaves differently in VAD vs STT turn detection mode

[MonkeyLeeT]

Bug Description

In 1.3.7, endpointing waits until last_speaking_time + min_endpointing_delay.

• VAD mode: last_speaking_time is set during speech (VAD inference), so by VAD END_OF_SPEECH the delay may already
  be satisfied → effective delay ≈ max(VAD_min_silence, min_endpointing_delay).
• STT mode: last_speaking_time is set at STT END_OF_SPEECH, so the full min_endpointing_delay is always added after
  STT EOS → additive.

Docs imply a consistent “after EOS” delay, but behavior is mode‑dependent and EOU metrics in STT mode exclude STT
silence which is perceived by user.

Expected Behavior

Either:

• Consistent “after end‑of‑utterance signal” delay in all modes, or
• Docs/metrics clearly state VAD = max(), STT = additive.

Reproduction Steps

1. VAD mode: set VAD min_silence=2s, min_endpointing_delay=2s. Observe commit ≈ 2s after user stop, not 4s.
2. STT mode (with providers like AssemblyAI): set STT EOS silence=2s and min_endpointing_delay=2s. Observe total ≈ 4s (STT silence + endpointing), while EOU metric end_of_utterance_delay shows ~2s.

Operating System

Linux

Models Used

No response

Package Versions

livekit-agents=1.3.7

Session/Room/Call IDs

No response

Proposed Solution

- Update docs to explain mode‑dependent behavior.
- Or set last_speaking_time at EOS in VAD mode for consistent additive behavior.
- Add a metric for “silence‑to‑commit” including STT EOS delay.

Additional Context

No response

Screenshots and Recordings

No response

[longcw]

min_endpointing_delay is the delay after the end of user speech, it's the same for both VAD and STT mode. the difference is with VAD we have a more accurate timing when the user stopped speaking (before VAD fires END_OF_SPEECH event).

Docs imply a consistent “after EOS” delay

can you point me where this is mentioned in the doc?

[MonkeyLeeT]

min_endpointing_delay is the delay after the end of user speech, it's the same for both VAD and STT mode. the difference is with VAD we have a more accurate timing when the user stopped speaking (before VAD fires END_OF_SPEECH event).

Docs imply a consistent “after EOS” delay

can you point me where this is mentioned in the doc?

agents/livekit-agents/livekit/agents/voice/agent_session.py

Line 210 in a5043a0

min_endpointing_delay (float): Minimum time-in-seconds the agent

Like I mentioned, due to how we set last_speaking_time differently in vad/tts mode, min_endpointing_delay means differently, in a practical manner. Specifically, TTS mode will set it

agents/livekit-agents/livekit/agents/voice/audio_recognition.py

Line 452 in a5043a0

self._last_speaking_time = time.time()

here, while VAD does it differently.

And my repro steps show that clearly users should have different consideration when tuning params.

[longcw]

min_endpointing_delay (float): Minimum time-in-seconds the agent
must wait after a potential end-of-utterance signal (from VAD
or an EOU model) before it declares the user’s turn complete.
Default 0.5 s.

I see, "a potential end-of-utterance signal" for VAD is still the EOS event, but we use a more accurate timestamp where user stopped speaking. we won't want to wait for VAD_min_silence + min_endpointing_delay which doubles the latency. For STT, it's just a technical limitation that we cannot know the exact time user stopped speaking but just the time the EOS event fired from STT.

feel free to create a pr to update the doc here.