support agent.say in before_llm_cb

[longcw]

This pr supports calling agent.say inside a before_llm_cb, it will playout the content of say first then the returned llm stream

async def before_llm_cb(
    agent: VoicePipelineAgent, chat_ctx: llm.ChatContext
) -> llm.LLMStream:
    speech_handle = await agent.say("say something")

    llm_stream = agent.llm.chat(chat_ctx=chat_ctx, fnc_ctx=agent.fnc_ctx)
    # await speech_handle.join()  # optionally wait for the playout finished
    return llm_stream

[changeset-bot]

🦋 Changeset detected

Latest commit: 8ee3b09

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
livekit-agents	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jayeshp19]

Should we wrap playing_lock.release() in finally block?
something like

await playing_lock.acquire()
try:
    ...(existing logic)
finally:
    playing_lock.release()

[longcw]

it might be okay as if the main speech failed we should stop the whole speech handle and its nested speeches.

[taqiycarbon]

Nice mate ! Took me a while to figure this out.

I went a different route and since before_llm_cb is always called after user_stopped_speaking I create the handle there.

   async def delayed_starter_words_generator(event, agent):
        await event.wait()

        starter_words = None
        if agent.starter_words:
            starter_words = agent.starter_words
            agent.starter_words = None

        yield starter_words  # Then yield the response

    async def say_starter_words(agent):
        start_time = time.perf_counter()
        event = asyncio.Event()
        gen = delayed_starter_words_generator(event, agent)
        # Store the event on the agent to access it in before_llm_cb
        agent.starter_words_event = event
        await agent.say(gen)

and then all you need to do is

async def before_llm_cb(agent, chat_ctx):
    logger.debug("Entering before_llm_cb.")


    if hasattr(agent, 'starter_words_event'):
        filler = random.choice(FILLER_OPTIONS)
        agent.starter_words = filler
        agent.starter_words_event.set()
        agent.starter_words_event = None

For those scared that it might block their queue you can add fallbacks to force release.

I put this there in case the PR is not approved for people that would face this problem and are considering writing monkey patches 🐒

Otherwise this is an important feature and it would be great to have it as part of the official SDK 👽