Simplismart Integration in Livekit

[Tushar-ml]

This is an integration PR of Simplismart's open source stt, tts and LLM models

[CLAassistant]

All committers have signed the CLA.

[Hormold]

Hello and thanks for the PR! How can we test it? I checked your web app and can't find any TTS models or endpoints to test with.
I've had success with public big Whisper-v3, and it works pretty fast 💨

[Tushar-ml]

sure @Hormold , we are releasing it by today, and let you know

[simplipratik]

Hey @Hormold,
You can use this code snippet to test the integration. As @Tushar-ml said, Livekit compatible TTS models will be released on Simplismart platform today as well, if you'd prefer that.

Happy to help if you have any more questions.

Example code snippet:

from livekit.agents import (
    Agent,
    AgentSession,
    JobContext,
    RunContext,
    WorkerOptions,
    cli,
    function_tool,
)
from livekit.plugins import silero
from livekit.plugins import simplismart, openai
from dotenv import load_dotenv
import os

load_dotenv()

SIMPLISMART_API_KEY = os.getenv("SIMPLISMART_API_KEY")

@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""

    return {"weather": "sunny", "temperature": 70}


async def entrypoint(ctx: JobContext):
    await ctx.connect()

    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
    )
    session = AgentSession(
        vad=silero.VAD.load(),
        # any combination of STT, LLM, TTS, or realtime API can be used        
        stt=simplismart.STT(base_url="https://api.simplismart.live/predict", api_key=SIMPLISMART_API_KEY, model="openai/whisper-large-v3-turbo"),
        llm=openai.LLM(model="google/gemma-3-4b-it", api_key=SIMPLISMART_API_KEY, base_url="https://api.simplismart.live"),
        tts=simplismart.TTS(base_url="https://api.simplismart.live/tts", api_key=SIMPLISMART_API_KEY, model="Simplismart/orpheus-3b-0.1-ft"),
    )

    await session.start(agent=agent, room=ctx.room)


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

[Hormold]

I tested it, and the voice sounds pretty smooth! The LLM isn't working for me; it's stuck with Gemma (7b and 28b) and Llama (70b). It works, but it's very, very slow.

[simplipratik]

@Hormold Thanks for testing this out, glad to hear you liked the voice quality!

For the LLM latency, I’d recommend trying a smaller model like Gemma 3 4B (google/gemma-3-4b-it). Larger models can have higher TTFT, which is likely what you’re seeing. I’ll also dig deeper on our side to see if there’s anything we can further optimize.

In the meantime, if the STT and TTS components look good, could we go ahead and merge this PR?

Merry Christmas in advance 🎄