Awaiting AgentTask in tool deadlocks

[Bnowako]

Bug Description

Summary

Sometimes when a function tool attempts to await an AgentTask, RuntimeError is raised about circular wait on SpeechHandle.wait_for_playout(). The error message suggests using RunContext.wait_for_playout() instead, I think this does not help, as the await in the AgentTask impl

Traceback (most recent call last):
  File "/.uv/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File "/.uv/.venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File "/.uv/.venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 454, in start_as_current_span
    yield span
  File "/root/auri/capabilities/call_me_later/tool.py", line 55, in handler
    result = await LanguageAwareCallMeLaterTask(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/agent.py", line 779, in __await_impl
    await session._update_activity(self, previous_activity="pause", blocked_tasks=blocked_tasks)
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/agent_session.py", line 1055, in _update_activity
    await self._activity.pause(blocked_tasks=blocked_tasks or [])
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/agent_activity.py", line 678, in pause
    await self._close_session()
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/agent_activity.py", line 720, in _close_session
    await self._interrupt_paused_speech(old_task=self._interrupt_paused_speech_task)
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/agent_activity.py", line 2703, in _interrupt_paused_speech
    await self._paused_speech.interrupt()  # ensure the speech is done
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/speech_handle.py", line 158, in _await_impl
    await self.wait_for_playout()
  File "/.uv/.venv/lib/python3.12/site-packages/livekit/agents/voice/speech_handle.py", line 147, in wait_for_playout
    raise RuntimeError(
RuntimeError: cannot call `SpeechHandle.wait_for_playout()` from inside the function tool `callMeLater` that owns this SpeechHandle. This creates a circular wait: the speech handle is waiting for the function tool to complete, while the function tool is simultaneously waiting for the speech handle.
To wait for the assistant’s spoken response prior to running this tool, use `RunContext.wait_for_playout()` instead.

Use Case

We have a tool that needs to:

1. Hand control to a Task
2. Wait for the task to complete and get the result
3. Use the result to decide what to return from the tool

This is different from a simple agent handoff because we need the result back in the original tool context.
Current Code (fails)

@function_tool
async def handler(raw_arguments: dict[str, object], context: RunContext):
    result = await Task()
    
    if result.foo is None:
        raise ToolError("Whatever")
    
    return "Task finished successfully"

Questions

1. Is running an AgentTask from a function tool supported? The docs show tasks being run from on_enter, not from tools, but we've found examples in lk code
2. If not, what's the recommended pattern?

Environment

• livekit-agents: 1.3.9
• Python 3.12
  Related Docs

Expected Behavior

Awaiting task in tool never throws

Reproduction Steps

1. Implement agent task
2. In the main agent implement tool that awaits the task
3. (very hard to reproduce, we see it only on prod, never were able to repro)
...

Current Code (fails)

@function_tool
async def handler(raw_arguments: dict[str, object], context: RunContext):
    result = await Task()
    
    return "tool called succesfully"

Operating System

debian

Models Used

gpt, nova-2, sonic-3

Package Versions

"livekit-agents[codecs, images]==1.3.9",
    "livekit-plugins-elevenlabs==1.3.9",
    "livekit-plugins-gladia==1.3.9",
    "livekit-plugins-cartesia==1.3.9",
    "livekit-plugins-deepgram==1.3.9",
    "livekit-plugins-noise-cancellation~=0.2",
    "livekit-plugins-silero==1.3.9",
    "livekit-plugins-turn-detector==1.3.9",
    "livekit-plugins-openai==1.3.9",

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[tinalenguyen]

Hey @Bnowako, thanks for the detailed report! Awaiting AgentTask in a function tool is supported and shouldn't throw those errors. I wonder if you would still experience those errors after upgrading to the latest version of livekit-agents

Could you share a snippet of LanguageAwareCallMeLaterTask()? I'm wondering if potentially calling generate_reply() in on_enter complicated things

[Bnowako]

@tinalenguyen Good call, we are calling generate_reply() in on_enter but from the stack trace i understood that this is from overwritten __await__ impl of the AgentTask, which runs before on_enter.

    @tracer.start_as_current_span("call_me_later_task.enter")
    async def on_enter(self) -> None:
        span = trace.get_current_span()
        span.set_attribute("app.task.type", "call_me_later")
        logger.info("CallMeLaterTask on_enter")
        span.set_attribute("app.task.scheduled_datetime", self._scheduled_datetime.isoformat() if self._scheduled_datetime else "")

        self.session.generate_reply()

Unfortunately we have sometimes problem with missing spans, but neither span or log is present in our o11y tools, so I guess it throws before on_enter

[Bnowako]

Hey @tinalenguyen just bumped livekit and will observe if this still happens :)

[longcw]

perhaps related to #4586, @Bnowako is there a speech alongside the tool call?

[Bnowako]

@tinalenguyen it's still happening after upgrading lk to latest
@longcw no, that was our first assumption, so we added await ctx.wait_for_playout() but did not work (screen attached of the llm response that triggered tool with task that raised)

[Bnowako]

Just double checked, interrupted is set to false for this turn so I guess it's something else :'( it seems like the agent can't recover afterwards

[Bnowako]

@longcw I checked the PR now and I think it could also solve this issue. I guess there is race condition and it could still happen that SpeechHandle is not yet done even without content in tool call. It's happening 100% of times when the content is there, but also possible that it happens when there is no content.

When are you planning to release it?

[longcw]

@Bnowako we are going to lock the speech to be not interruptible when it has a tool awaiting for an AgentTask. it should be released soon in one or two days.