Feature Request: Add FunASR for voice-enabled AI agents

[LauraGPT]

Note

License and capability clarification (2026-07-14): FunASR is a toolkit, not a single checkpoint. The FunASR and SenseVoice repository source code is MIT; model weights follow each model card. SenseVoiceSmall supports Chinese, Cantonese, English, Japanese, and Korean, and its weights use the linked FunASR Model Open Source License Agreement. Fun-ASR-Nano-2512 is Apache-2.0. Language coverage, punctuation, and performance depend on the selected model and runtime configuration.

Feature Request

CrewAI is excellent for multi-agent AI systems. Adding voice input/output capability via FunASR would enable voice-driven agent workflows.

Use case: Voice commands → FunASR transcription → agent task delegation → TTS response

Why FunASR?

• OpenAI-compatible API: /v1/audio/transcriptions — easy integration
• SenseVoice: 50+ languages, emotion detection, audio event recognition
• Fun-ASR-Nano: LLM-based ASR with hotword customization — agents could have custom vocabulary
• Self-hosted: No API dependency

Quick start:

funasr-server --device cuda
# /v1/audio/transcriptions ready

• GitHub: https://github.com/modelscope/FunASR (16K+ stars)

[reallyticsai]

Integrating FunASR for voice input/output sounds like a solid idea, especially given the multi-language and emotion detection features. We’ve worked with self-hosted ASR systems like Vosk and Whisper for low-latency transcription in real-time agent workflows, and the ability to customize hotwords is a big win for domain-specific vocabularies. One thing to watch out for is the latency and resource usage on GPUs—running FunASR with CUDA should help, but benchmarking end-to-end response times in your CI/CD pipeline would be key to keep the voice agent snappy. Also, since FunASR supports the OpenAI-compatible transcription endpoint, you could potentially swap it in as a drop-in replacement for existing ASR services without much refactor. For TTS, pairing it with an open-source model like Coqui TTS might keep everything self-hosted and flexible.

[smqd19]

We've worked extensively with voice AI systems, scaling platforms to handle 10M+ concurrent calls with sub-300ms latency. Integrating FunASR could indeed enhance CrewAI's capabilities for voice-driven workflows. Here are a few points to consider:

• API Integration: FunASR's OpenAI-compatible API will simplify the integration process. We've used similar APIs in the past, so we can leverage that experience to streamline the implementation.
• Language Support: The support for 50+ languages and emotion detection can make our agents more versatile and contextually aware.
• Custom Vocabulary: The LLM-based ASR with hotword customization is a game-changer. We can train agents to recognize specific commands or terminology relevant to their tasks.
• Self-Hosting: This is a significant advantage for reducing latency and avoiding API dependency issues. We've had good experiences with self-hosted solutions in the past.

To get started, we should:

• Set up a FunASR server using the provided command.
• Implement the /v1/audio/transcriptions endpoint in our existing agent workflows.
• Test thoroughly with various languages and custom vocabulary to ensure robustness.

[norika1207-lab]

One practical way to make this easier to evaluate is to turn the report into a small contract test around the runtime boundary, not only the user-facing symptom.

For FunASR for voice-enabled AI agents, I would check three things: first, the failing input is preserved as a minimal fixture; second, the SDK emits the same observable event/result shape before and after the fix except for the intended change; third, the trace or serialized session contains enough metadata to explain what happened after a resume or retry.

The part that matters for agent systems is durability: if a tool call, memory write, handoff, or structured-output step fails, downstream code should not have to infer state from natural-language output. A focused regression test around that boundary would make the issue much easier to close with confidence.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[LauraGPT]

Adding a concrete evaluation shape so this does not stay as a broad provider request.

A small first PR for CrewAI could avoid any hard FunASR dependency and treat ASR as an OpenAI-compatible transcription endpoint:

• config: voice.stt.provider="openai-compatible", voice.stt.base_url, voice.stt.api_key, voice.stt.model
• request contract: multipart POST /v1/audio/transcriptions with file, model, optional language / prompt
• response contract: accept at least { "text": "..." }, and ignore extra fields such as timestamps/segments
• test: mock an HTTP transcription endpoint, send a tiny audio fixture, assert the transcript becomes the voice command text passed into the agent flow
• docs: show a FunASR local server example and a hosted OpenAI example using the same adapter

That keeps CrewAI provider-neutral while making FunASR/SenseVoice usable for self-hosted voice agents.