Exploring "linear attention" for text-to-speech.
It predicts audio codec "à la" MusicGen : delayed residual vector quantizers so that we do not need multiple models.
Featuring RWKV, Mamba, Gated Linear Attention.
Compared to other LM TTS model :
- Can be easily pretrained and finetuned on midrange GPUs.
- Tiny memory footprint.
- Trained on long context (up to 2000 tokens : ~27s).
| Model | #Params | Dataset | Checkpoint | Steps | Note |
|---|---|---|---|---|---|
| GLA | 60M, 130M | Librilight-medium | Download | 300k | GPU inference only |
| Mamba | 60M | Librilight-medium | Download | 300k | GPU inference only |
| RWKV v6 | 60M | LibriTTS | Download | 150k | GPU inference only |
Following the linear complexity LM you choose, follow respective instructions first:
- For Mamba check the official repo.
- For GLA/RWKV inference check flash-linear-attention.
- For RWKV training check RWKV-LM
Download configuration and weights above, then check Inference.ipynb.
- Fix RWKV6 inference and/or switch to FLA implem.
- Provide a Datamodule for training (lhotse might also work well).
- Implement CFG.
- Scale up.
- The RWKV authors and the community around for carrying high-level truly opensource research.
- @SmerkyG for making my life easy at testing cutting edge language model.
- @lucidrains for its huge codebase.
- @sustcsonglin who made GLA and FLA.
- @harrisonvanderbyl fixing RWKV inference.
@software{lemerle2024linaspeech,
title = {LinaSpeech: Exploring "linear attention" for text-to-speech.},
author = {Lemerle, Théodor},
url = {https://github.com/theodorblackbird/lina-speech},
month = april,
year = {2024}
}This work is performed in the Analysis/Synthesis team of the STMS Laboratory at IRCAM, and is part of the following project: ANR Exovoices
