🎙️ X-ASR

Streaming-focused automatic speech recognition models based on icefall/k2, Zipformer, and sherpa-onnx.

_{Participating Institutions}

🌐 中文版

📄 X-ASR-zh-en Technical Report: Coming Soon

🔍 Overview

🧩 X-ASR

X-ASR is a series of automatic speech recognition models built with the icefall framework. The series focuses on streaming ASR and low-latency deployment, while also supporting offline recognition. This repository currently releases an initial batch of Chinese-English streaming ASR models, and the X-ASR series will be continuously maintained, updated, and scaled across languages, model architectures, and training data.

🤖 X-ASR-zh-en

X-ASR-zh-en is trained on approximately 1 million hours of open-source and collected speech data. It is designed as an offline-streaming unified transducer ASR model with the Zipformer architecture, supporting both offline decoding and true streaming decoding. The model provides multiple streaming chunk sizes: 160 ms, 480 ms, 960 ms, and 1920 ms, supports punctuation and casing, and can be conveniently deployed with sherpa-onnx.

📅 Timeline

Status	Item	Details
✅ Released	`X-ASR-zh-en` initial release	Chinese-English offline-streaming unified ASR models, sherpa-onnx deployment artifacts, and online demo are available.
📄 Coming Soon	`X-ASR-zh-en` technical report	Training recipe, model architecture, evaluation protocol, deployment details, and ablation analysis will be released.
🌏 Upcoming	Thai, Indonesian, and Vietnamese ASR	Streaming ASR models for the next language releases are under preparation.
🔄 Ongoing	Model and data updates	Continued work on model scaling, architecture improvements, data refinement, latency, stability, punctuation, and casing.

📦 Model Releases

Model	Languages	Type	Streaming chunks	Deployment	Report	Model files
`X-ASR-zh-en`	Chinese, English	Offline-streaming unified transducer ASR	160 ms, 480 ms, 960 ms, 1920 ms	sherpa-onnx	Coming Soon	GitHub, Hugging Face, ModelScope

⭐ Highlights

Category	Description
Framework	icefall / k2
Architecture	Zipformer transducer
Training scale	Approximately 1 million hours of open-source and collected speech data
Current languages	Chinese and English
Decoding modes	Offline decoding and true streaming decoding
Streaming chunks	160 ms, 480 ms, 960 ms, 1920 ms
Text output	Supports punctuation and casing
Runtime	sherpa-onnx
Interface	WebSocket streaming server and WAV-file client

🎙️ Applications

We welcome more experiments and real-world use cases built on top of X-ASR. The following downstream applications are based on X-ASR and have been synced into this repository.

🧪 Vibe-Coding Application with FireRedVAD

X-ASR local offline live recognition demo

Local Offline Vibe-Coding ASR Demo
_{Microphone/WAV → FireRedVAD endpointing → X-ASR streaming decoding → live partial/final output. Designed for local offline dictation, voice-input prototypes, and vibe-coding workflows.}

This application turns X-ASR from a model release into a complete local voice-input loop. FireRedVAD detects when speech starts and ends, while X-ASR performs low-latency streaming recognition during the utterance. A short pause commits the current sentence as final text.

The main idea is that streaming ASR alone is not enough for interactive use: the decoder can produce partial text, but it does not know when a user has finished speaking. Adding VAD-based endpointing makes the system usable for local dictation, voice-IME prototypes, and vibe-coding scenarios where speech can be turned into text without sending audio to a server.

As a starting point, the demo prints final results in the terminal. A natural next step is to replace that final-text callback with an editor or focused-input injection layer, turning X-ASR into a local hands-free coding and writing interface.

Open Guide · 中文

⬇️ Desktop Package Download

Vibe XASR · a local voice input method powered by X-ASR

⬇️ Download for macOS →
_{Universal (Apple Silicon + Intel) · macOS 15.0+ · signed & notarized · auto-updates in-app}

Hold a hotkey, speak, and the text lands right at your cursor — 100% local & offline, your data never leaves the device. The X-ASR streaming engine turns Chinese & English speech (freely code-switched) into text in real time, system-wide.

Core features

🎙️ Three dictation modes — insert-on-finish · live streaming (types as you talk) · OnCall standby (floating window)
📋 Built-in pad & history — saved by date; copy / edit / export
📖 Personal dictionary — hotwords, homophone correction, replace rules
✨ AI Polish (Beta) — optional cloud LLM smooths wording & removes fillers (opt-in, off by default)
🔒 Privacy-first & auto-update — fully offline; one-click upgrades inside the app

_{🪟 A Windows build is also available (in Releases) — an early preview, not yet fully tested, kept in sync with the latest macOS features. Please report issues as you run into them.}

📊 Evaluation

The following results are for the current X-ASR-zh-en release. All results are reported with greedy search. Measurement: English results use WER (%), and Chinese results use CER (%); lower is better.

🧪 Public ASR Benchmarks

⚙️ Mode	⏱️ Chunk size	📚 LibriSpeech		🎙️ GigaSpeech	🗣️ WenetSpeech
⚙️ Mode	⏱️ Chunk size	clean	other	🎙️ GigaSpeech	net	meeting
Streaming	160 ms	3.49	8.75	10.32	8.72	10.47
Streaming	480 ms	2.99	7.36	9.70	7.46	9.11
Streaming	960 ms	2.87	6.77	9.59	6.97	8.40
Streaming	1920 ms	2.75	6.33	9.43	6.58	7.88
Offline	-	2.56	5.56	9.17	5.83	7.06

Note: Bold numbers indicate the best result among the listed modes for each benchmark column.

🏆 Public Benchmark Model Comparison

The following table compares representative ASR models on the same public benchmark columns. Ranks are computed by AVG across the five listed columns; lower is better. Parameter sizes are shown when provided by the source sheet.

🏅 Rank	Model	Params	📚 LibriSpeech		🎙️ GigaSpeech	🗣️ WenetSpeech		AVG
🏅 Rank	Model	Params	clean	other	🎙️ GigaSpeech	net	meeting	AVG
1	Qwen3-ASR	1.7B	1.65	3.45	8.56	5.29	5.46	4.882
2	Qwen3-ASR	0.6B	2.18	4.54	8.94	5.97	6.88	5.702
3	X-ASR-zh-en (offline)	0.16B	2.56	5.56	9.17	5.83	7.06	6.036
4	SenseVoice-small	234M	3.16	7.21	11.24	5.73	6.47	6.762
5	VibeVoice-ASR	9B	2.18	5.65	9.49	14.45	17.19	9.792

🧭 Vertical-Domain Benchmarks

The following results report GigaSpeechBench vertical-domain performance for the current X-ASR-zh-en release. Values are WER/CER percentages; lower is better. Domain abbreviations follow the GigaSpeechBench vertical-domain labels.

CH

⚙️ Mode	⏱️ Chunk size	ARG	AIT	ART	BIO	ECM	ENG	ENT	FIN	HUM	LAW	MED	MIL
Streaming	160 ms	9.88	6.76	4.39	7.32	4.13	3.58	8.45	3.23	10.42	6.58	4.25	2.55
Streaming	480 ms	8.67	6.17	3.60	6.22	3.78	3.04	7.04	2.78	9.43	5.84	3.76	2.11
Streaming	960 ms	8.00	5.69	3.44	6.10	3.69	2.88	6.71	2.72	9.07	5.58	3.69	2.11
Streaming	1920 ms	7.24	5.58	3.27	5.82	3.48	2.74	6.55	2.57	8.59	4.97	3.53	1.94
Offline	-	6.56	4.54	2.77	5.04	2.99	2.32	6.02	1.94	7.64	4.20	2.90	1.68

EN

⚙️ Mode	⏱️ Chunk size	ARG	AIT	ART	BIO	ECM	ENG	ENT	FIN	HUM	LAW	MED	MIL
Streaming	160 ms	5.29	8.57	8.55	7.31	4.33	5.01	16.25	5.58	7.36	13.39	6.03	6.20
Streaming	480 ms	4.62	8.40	7.73	6.12	4.19	4.65	14.50	5.21	6.79	11.51	5.59	6.02
Streaming	960 ms	4.58	8.35	7.45	6.00	4.13	4.44	13.99	5.12	6.58	10.86	5.52	6.04
Streaming	1920 ms	4.33	8.32	6.90	5.89	4.00	4.37	13.61	4.98	6.39	10.52	5.45	5.78
Offline	-	4.09	8.28	6.73	5.48	4.12	4.30	12.30	4.94	6.17	10.41	5.35	5.61

🎧 Demo

A sherpa-onnx based online demo is available here:

https://stream-asr.sjtuxlance.com/

Demo video:

Open demo video

🚀 Quick Start

This section shows how to build and run the sherpa-onnx WebSocket streaming server and the corresponding WebSocket client. For complete deployment arguments, model switching, runtime options, and production notes, see the deployment guide.

1. Clone or download model artifacts

This repository uses Git LFS for ONNX model artifacts and demo media. Install Git LFS before cloning or before pulling large files.

GitHub

Use GitHub when you want the full project repository, bilingual documentation, training references, deployment examples, and issue-tracking context.

git lfs install
git clone https://github.com/Gilgamesh-J/X-ASR.git
cd X-ASR
git lfs pull

Hugging Face

Use Hugging Face when you want the model artifact page and standard HF Hub download tooling.

hf download GilgameshWind/X-ASR-zh-en \
  --local-dir ./X-ASR-zh-en

ModelScope

Use ModelScope when you prefer the ModelScope mirror or Git LFS clone from ModelScope.

git lfs install
git clone https://www.modelscope.ai/Gilgamesh-J/X-ASR-zh-en.git
cd X-ASR-zh-en
git lfs pull

2. Prepare the sherpa-onnx runtime

If you cloned the full GitHub project, enter:

cd X-ASR/X-ASR-zh-en/deployment

If you downloaded from Hugging Face or cloned from ModelScope, enter:

cd X-ASR-zh-en/deployment

Then prepare the Python environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

3. Start the WebSocket server

The server wraps sherpa_onnx.OnlineRecognizer and exposes a WebSocket endpoint. Each WebSocket connection keeps an independent recognizer session, so concurrent clients do not share decoding state. The example below starts the 160 ms streaming model on CPU and listens on ws://0.0.0.0:6666.

python infer_and_client/sherpa_streaming_server.py \
  --host 0.0.0.0 \
  --port 6666 \
  --tokens models/chunk-160ms-model/tokens.txt \
  --encoder models/chunk-160ms-model/encoder-160ms.onnx \
  --decoder models/chunk-160ms-model/decoder-160ms.onnx \
  --joiner models/chunk-160ms-model/joiner-160ms.onnx \
  --provider cpu \
  --sample-rate 16000 \
  --feature-dim 80 \
  --num-threads 1 \
  --decoding-method greedy_search \
  --model-type zipformer2 \
  --enable-endpoint-detection 0 \
  --text-format none

The --tokens, --encoder, --decoder, and --joiner files must come from the same model directory.

Optional: interactive tail-probe mode

The default mode decodes only from audio chunks received from the client. For interactive voice-input scenarios, you can enable the optional energy-based tail probe. After speech activity is observed and low-energy audio lasts long enough, the server appends a short silence segment and runs decoding again. This can help trailing partial results refresh after the user pauses.

python infer_and_client/sherpa_streaming_server.py \
  --host 0.0.0.0 \
  --port 6666 \
  --tokens models/chunk-160ms-model/tokens.txt \
  --encoder models/chunk-160ms-model/encoder-160ms.onnx \
  --decoder models/chunk-160ms-model/decoder-160ms.onnx \
  --joiner models/chunk-160ms-model/joiner-160ms.onnx \
  --provider cpu \
  --sample-rate 16000 \
  --feature-dim 80 \
  --num-threads 1 \
  --decoding-method greedy_search \
  --model-type zipformer2 \
  --enable-endpoint-detection 0 \
  --text-format none \
  --enable-energy-tail-probe 1 \
  --low-energy-rms 0.003 \
  --speech-rms 0.010 \
  --min-speech-ms 200 \
  --min-silence-ms 500 \
  --tail-probe-ms 500 \
  --tail-probe-cooldown-ms 1000

Tune the RMS and silence thresholds according to microphone gain, background noise, and frontend audio chunking behavior. Keep --enable-energy-tail-probe 0 for the most conservative server behavior.

4. Run the WebSocket client

Open another terminal:

cd X-ASR-zh-en/deployment
source .venv/bin/activate

python infer_and_client/sherpa_streaming_client.py \
  --server-uri ws://127.0.0.1:6666 \
  --wav /path/to/test.wav \
  --chunk-ms 100 \
  --simulate-realtime 1

The client loads a WAV file, converts or resamples it to 16 kHz mono int16 PCM, sends binary PCM chunks over WebSocket, and prints partial/final recognition results returned by the server. With --simulate-realtime 1, --chunk-ms 100 means one audio packet is sent roughly every 100 ms.

5. WebSocket protocol

The provided client and server use a minimal streaming protocol:

Step	Message	Purpose
1	JSON: `{"type": "start", "sample_rate": 16000}`	Start one recognition session
2	Binary: int16 PCM audio chunks	Stream audio to the recognizer
3	JSON: `{"type": "end"}`	Finish the session and flush final results

For detailed deployment instructions, see X-ASR-zh-en/deployment/README.md. For the script-level server/client guide and full parameter reference, see X-ASR-zh-en/deployment/infer_and_client/README.md.

🗂️ Repository Layout

X-ASR/
|-- README.md
|-- README_zh.md
|-- LICENSE
|-- assets/
|   |-- figures/
|   |   |-- demo-preview.png
|   |   `-- zipformer.png
|   |-- demos/
|   |   `-- demo.mov
|   `-- institutions/
|       |-- sjtu.png
|       |-- sii.png
|       |-- fudan.png
|       `-- hust.png
`-- X-ASR-zh-en/
    |-- deployment/
    |   |-- README.md
    |   |-- requirements.txt
    |   |-- infer_and_client/
    |   |   |-- README.md
    |   |   |-- sherpa_streaming_infer.py
    |   |   |-- sherpa_streaming_server.py
    |   |   `-- sherpa_streaming_client.py
    |   |-- x-asr-live-demo/
    |   |   |-- README.md
    |   |   |-- README_zh.md
    |   |   |-- live_asr.py
    |   |   |-- download_models.sh
    |   |   |-- requirements.txt
    |   |   `-- assets/
    |   `-- models/
    |       |-- README.md
    |       |-- chunk-160ms-model/
    |       |-- chunk-480ms-model/
    |       |-- chunk-960ms-model/
    |       `-- chunk-1920ms-model/
    `-- zipformer/
        |-- README.md
        |-- train.py
        |-- finetune.py
        |-- decode.py
        |-- streaming_decode.py
        |-- export.py
        |-- export-onnx.py
        |-- export-onnx-streaming.py
        |-- model.py
        |-- zipformer.py
        |-- data/
        |   |-- lang_5000/
        |   |   |-- bpe.model
        |   |   `-- tokens.txt
        |   `-- lang_5000_with_punctuation/
        |       |-- bpe_punc.model
        |       `-- tokens.txt
        `-- checkpoint/
            |-- pretrained.pt
            `-- fintuned_with_punctuation.pt

X-ASR-zh-en/deployment/ contains runnable sherpa-onnx deployment artifacts, including the WebSocket server/client path and the local live ASR application demo. X-ASR-zh-en/zipformer/ contains the icefall/Zipformer training, decoding, export recipe files, tokenizer/data files, and released PyTorch checkpoints for the model.

🤝 Contributing

We welcome feedback and contributions in the following areas:

Deployment issues on different CPU/GPU environments
Streaming latency and stability reports
Evaluation results on new datasets or domains
Requests for new languages or future releases
Improvements to documentation and examples

When reporting deployment problems, please include the environment, command, input audio format, and error log.

📜 License

This project is released under the Apache-2.0 License.

🙏 Acknowledgements

This model series is trained with icefall and deployed with sherpa-onnx.

icefall: https://github.com/k2-fsa/icefall
sherpa-onnx: https://github.com/k2-fsa/sherpa-onnx

Navigation Menu

Repository files navigation

🎙️ X-ASR

🔍 Overview

🧩 X-ASR

🤖 X-ASR-zh-en

📅 Timeline

📦 Model Releases

⭐ Highlights

🎙️ Applications

🧪 Vibe-Coding Application with FireRedVAD

⬇️ Desktop Package Download

📊 Evaluation

🧪 Public ASR Benchmarks

🏆 Public Benchmark Model Comparison

🧭 Vertical-Domain Benchmarks

CH

EN

🎧 Demo

🚀 Quick Start

1. Clone or download model artifacts

GitHub

Hugging Face

ModelScope

2. Prepare the sherpa-onnx runtime

3. Start the WebSocket server

Optional: interactive tail-probe mode

4. Run the WebSocket client

5. WebSocket protocol

🗂️ Repository Layout

🤝 Contributing

📜 License

🙏 Acknowledgements

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
X-ASR-zh-en		X-ASR-zh-en
assets		assets
projects		projects
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README_zh.md		README_zh.md

Latest commit

History

Folders and files

Repository files navigation

🎙️ X-ASR

🔍 Overview

🧩 X-ASR

🤖 X-ASR-zh-en

📅 Timeline

📦 Model Releases

⭐ Highlights

🎙️ Applications

🧪 Vibe-Coding Application with FireRedVAD

⬇️ Desktop Package Download

📊 Evaluation

🧪 Public ASR Benchmarks

🏆 Public Benchmark Model Comparison

🧭 Vertical-Domain Benchmarks

CH

EN

🎧 Demo

🚀 Quick Start

1. Clone or download model artifacts

GitHub

Hugging Face

ModelScope

2. Prepare the sherpa-onnx runtime

3. Start the WebSocket server

Optional: interactive tail-probe mode

4. Run the WebSocket client

5. WebSocket protocol

🗂️ Repository Layout

🤝 Contributing

📜 License

🙏 Acknowledgements

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages