Add MOSS-TTSD with XY-Tokenizer by gaoyang07 · Pull Request #40286 · huggingface/transformers

gaoyang07 · 2025-08-19T14:54:30Z

What does this PR do?

MOSS-TTSD 🪐

Overview

MOSS-TTSD (text to spoken dialogue) is an open-source bilingual spoken dialogue synthesis model that supports both Chinese and English. It can transform dialogue scripts between two speakers into natural, expressive conversational speech. MOSS-TTSD supports voice cloning and single-session speech generation of up to 960 seconds, making it ideal for AI podcast production.

Highlights

Highly Expressive Dialogue Speech: Built on unified semantic-acoustic neural audio codec, a pre-trained large language model, millions of hours of TTS data, and 400k hours synthetic and real conversational speech, MOSS-TTSD generates highly expressive, human-like dialogue speech with natural conversational prosody.
Two-Speaker Voice Cloning: MOSS-TTSD supports zero-shot two speakers voice cloning and can generate conversational speech with accurate speaker swithcing based on dialogue scripts.
Chinese-English Bilingual Support: MOSS-TTSD enables highly expressive speech generation in both Chinese and English.
Long-Form Speech Generation (up to 960 seconds): Thanks to low-bitrate codec and training framework optimization, MOSS-TTSD has been trained for long speech generation, enabling single-session speech generation of up to 960 seconds.
Fully Open Source & Commercial-Ready: MOSS-TTSD and its future updates will be fully open-source and support free commercial use.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Rocketknight1 · 2025-08-20T11:11:35Z

cc @eustlb

gaoyang07 · 2025-08-27T07:09:04Z

@ebezzam @eustlb Hey there, I think now the failed CI cases are not related to our PR changes, so maybe you can start reviewing on this? Thanks~

ebezzam · 2025-08-27T07:41:11Z

hey @gaoyang07 thanks a bunch for the PR! We'll try to get in an initial review by next week, we should have more bandwidth then! Thanks for your patience 🙏

gaoyang07 · 2025-09-09T12:58:25Z

hey @gaoyang07 thanks a bunch for the PR! We'll try to get in an initial review by next week, we should have more bandwidth then! Thanks for your patience 🙏

Hey @ebezzam , hope all is well~ Just checking in on this PR — no rush if you’re tied up, but I’d appreciate your review whenever you get a chance. Thanks~

ebezzam · 2025-09-09T16:23:05Z

Hi @gaoyang07 thanks for your message. I started the review and hope to give you some initial comments by the end of the week!

ebezzam

Thanks for the PR, a lot of things here that are great to add to Transformers and a very nice model to add to the collection. Thanks for open-sourcing and making the effort to contribute 🙏

I've done a first review to focus on the Tokenizer since MOSS-TTSD will depend on it + iterating on the tokenizer will help you get used to some Transformer conventions for us to then iterate on MOSS-TTSD itself more efficiently!

Some general comments:

We want to explicitly expose parameters in the configuration. So we want to avoid something like self.params = kwargs in the configuration. I've tried to point some examples to other codecs that may be useful for you.
We have this tool known as modular to reuse code from other models. As I see from your paper that you make use of Whisper's encoder and various attention modules, we will want to use modular to import/inherit from replicated components (e.g. Whisper, Llama, DAC), and optionally add more parameters to the init or overwrite methods like forward if a slight variation is needed (see this example). The modular script will be used to generate modeling_xy_tokenizer.py. For the review process, modular also help us to clearly see the new features/architectures that your model is adding to Transformers.
Could you also add a conversion script? Essentially it's the script that will map weights from the original model to the Transformers-compatible one, and upload the model and its configuration to the hub. You can see an example here.
I was a bit confused by your feature extractor. If your encoders follow the same architecture as Whisper, why can't you directly use Whisper's feature extractor?
Removing unused features / options / code branch / etc

Hope that helps and let me know if you have any questions!

ebezzam · 2025-09-09T14:32:12Z

+        return output_lengths
+
+
+class XYTokenizerFeatureExtractor(WhisperFeatureExtractor):


I haven't dug deep into how your feature extractors differs from WhisperFeatureExtractor, but is there a reason why the WhisperFeatureExtractor can't be used as is?

From what I understand in your paper (Sec 3.1 - Encoder), the semantic and acoustic encoder are initialized to Whisper's encoder, so I would expect the same feature extractor could be used?

And a new model can used an existing feature extractor, by configuring like so.

We implemented XYTokenizerFeatureExtractor mainly for two reasons:

In WhisperFeatureExtractor
, the maximum frequency is fixed at 8000, which works for 16 kHz inputs but may not generalize. Since we plan to support 24 kHz inputs in XY-Tokenizer, we needed more flexibility.

WhisperFeatureExtractor simply truncates inputs longer than 30s. In contrast, XYTokenizerFeatureExtractor splits them into consecutive 30s chunks and adds configurable overlap (right-sided or two-sided), so we can handle inference on audio longer than 30s.

So while they are very similar, we needed these extensions for XY-Tokenizer.

ebezzam · 2025-09-09T19:02:20Z

+        input_dim: int = 1280,
+        rvq_dim: Optional[int] = None,
+        output_dim: Optional[int] = None,
+        num_quantizers: int = 32,
+        codebook_size: int = 1024,
+        codebook_dim: int = 8,
+        quantizer_dropout: float = 0.5,
+        skip_rvq_ratio: float = 0.0,
+        vq_config: VectorQuantizerConfig = None,


Similar to other modules, let's explicitly store the relevant parameters in XYTokenizerConfig. You can how other codecs also specify codebook parameters in the configuration (DAC, EnCodec, Xcodec).

ebezzam · 2025-09-09T19:03:18Z

+        self.input_proj = WNConv1d(input_dim, rvq_dim, 1) if input_dim != rvq_dim else nn.Identity()
+        self.output_proj = WNConv1d(rvq_dim, self.output_dim, 1) if rvq_dim != self.output_dim else nn.Identity()


Let's remove weight norm as mentioned above.

ebezzam · 2025-09-09T19:06:12Z

+        return self.out_project(embeddings)
+
+
+class ResidualVQ(nn.Module):


Are there similarities with DAC's RVQ? If so, we can simularly use modular to inherit/import from it.

Thanks for pointing this out! 🙏
There are some similarities, but our quantization method is different from DAC’s. Specifically, as shown in DACVectorQuantize
, DAC normalizes both the input vectors and the codebook before the nearest-neighbor search, while in our implementation we do not apply this normalization step.

github-actions · 2025-12-12T08:32:03Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, moss_ttsd, xy_tokenizer

github-actions · 2025-12-12T08:40:31Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=40286&sha=8d9481

gaoyang07 changed the title ~~[Feature] Support MOSS-TTSD with XY-Tokenizer~~ [WIP] [Feature] Support MOSS-TTSD with XY-Tokenizer Aug 19, 2025

gaoyang07 changed the title ~~[WIP] [Feature] Support MOSS-TTSD with XY-Tokenizer~~ [WIP] [models] Add MOSS-TTSD with XY-Tokenizer Aug 19, 2025

ebezzam added the Audio label Aug 20, 2025

gaoyang07 force-pushed the moss-ttsd branch 4 times, most recently from 2eedcef to 0d23e64 Compare August 25, 2025 05:59

gaoyang07 changed the title ~~[WIP] [models] Add MOSS-TTSD with XY-Tokenizer~~ Add MOSS-TTSD with XY-Tokenizer Aug 25, 2025

gaoyang07 force-pushed the moss-ttsd branch 2 times, most recently from 1687d1a to a2d4c6e Compare August 26, 2025 11:11

ebezzam reviewed Sep 9, 2025

View reviewed changes

kaysonyu force-pushed the moss-ttsd branch 6 times, most recently from a95029f to 2cd6dc0 Compare December 10, 2025 08:39

gaoyang07 and others added 8 commits December 12, 2025 04:38

add MOSS-TTSD with XY_Tokenizer

175819f

refactor tests

343da5b

fix CI

c1fd86c

fix CI

a74c6b3

fix CI

4615105

fix CI

b2d0ee5

fix by comments

ac0dd3b

fix: fix _tied_weights_keys & tie_weights

5419e5b

kaysonyu added 5 commits December 12, 2025 04:38

chore: format code

fa9b49a

fix: rm tie_weights for rebase

a29c3b5

fix: fix codebook_dim

be0724a

fix: fix tp_plan

41662d4

fix: fix XYTokenizerConfig

ee9ee40

kaysonyu force-pushed the moss-ttsd branch from 842b05c to ee9ee40 Compare December 12, 2025 04:39

kaysonyu added 2 commits December 12, 2025 07:13

feat: expose parameters in config instead of using kwargs

52d1025

chore: ruff

8d94818

kaysonyu force-pushed the moss-ttsd branch from b581efe to 8d94818 Compare December 12, 2025 08:30

evalstate mentioned this pull request Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

		return output_lengths


		class XYTokenizerFeatureExtractor(WhisperFeatureExtractor):

		self.input_proj = WNConv1d(input_dim, rvq_dim, 1) if input_dim != rvq_dim else nn.Identity()
		self.output_proj = WNConv1d(rvq_dim, self.output_dim, 1) if rvq_dim != self.output_dim else nn.Identity()

		return self.out_project(embeddings)


		class ResidualVQ(nn.Module):

Uh oh!

Conversation

gaoyang07 commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

MOSS-TTSD 🪐

Overview

Highlights

Before submitting

Who can review?

Uh oh!

Rocketknight1 commented Aug 20, 2025

Uh oh!

gaoyang07 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebezzam commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaoyang07 commented Sep 9, 2025

Uh oh!

ebezzam commented Sep 9, 2025

Uh oh!

ebezzam left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ebezzam Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

gyt1145028706 Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

ebezzam Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

ebezzam Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ebezzam Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

gyt1145028706 Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Dec 12, 2025

Uh oh!

github-actions Bot commented Dec 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gaoyang07 commented Aug 19, 2025 •

edited

Loading

gaoyang07 commented Aug 27, 2025 •

edited

Loading

ebezzam commented Aug 27, 2025 •

edited

Loading

ebezzam left a comment •

edited

Loading