Clarification on ep_pitches, ep_notedurs,ep_types and required annotations for style transfer with custom dataset

[spike233666]

Hi, thank you for your great work on TCSinger.
I am currently preparing a new dataset for a style transfer task using TCSinger, and I would like to clarify a few details regarding the required input format for metadata.json. In your README, you mention the following fields:

“Put metadata.json (including ph, word, item_name, ph_durs, wav_fn, singer, ep_pitches, ep_notedurs, ep_types for each singing voice) and phone_set.json (all phonemes of your dictionary) in data/processed/tc...”

I have the following questions regarding these fields:

1. What do ep_pitches, ep_notedurs, and ep_types represent?
   I noticed that in GTSinger’s metadata.json, these arrays seem to match the length of ph, but ep_notedurs does not align with ph_durs. Could you please explain the meaning of the prefix ep_ and how these features relate to phonemes (ph)?how can I extract or obtain these features from my own dataset?
2. For style transfer only (e.g., zero-shot singing voice synthesis based on a reference audio), do I still need to provide other annotations like tech and emotion? Or are they only required for tasks like style control?

Thanks in advance for your help!

[AaronZ345]

1. "ep" stands for "each phoneme." It refers to replicating information to match the length of the phoneme list, making it easier for the model to learn and align. If a single note corresponds to multiple phonemes, the pitch and duration of the note are duplicated for each phoneme. This correspondence can be inferred from the score, as it typically includes word-to-note mappings. Since the duration values are copied, they do not match the actual phoneme durations. The model must learn to segment note durations and map them to true phoneme durations based on fine-grained musical timing (such as 1/16 notes).

2. Style transfer does not require additional annotations, provided you do not need explicit emotion labels as input.

[jayzen33]

1. "ep" stands for "each phoneme." It refers to replicating information to match the length of the phoneme list, making it easier for the model to learn and align. If a single note corresponds to multiple phonemes, the pitch and duration of the note are duplicated for each phoneme. This correspondence can be inferred from the score, as it typically includes word-to-note mappings. Since the duration values are copied, they do not match the actual phoneme durations. The model must learn to segment note durations and map them to true phoneme durations based on fine-grained musical timing (such as 1/16 notes).
2. Style transfer does not require additional annotations, provided you do not need explicit emotion labels as input.

Could you please explain what ep_types does? The structure of this repository is a bit unclear to me. A disorganized open-source project can be more challenging to use than having no code at all. I'd like to use your model as a baseline, but I'm finding it difficult to replicate your results. The documentation for the test set construction and the inference steps is unclear. Could you provide more details?

[AaronZ345]

Hi! Here’s a clearer overview of the repository structure and how to work with it, along with a refined explanation of ep_types and notes on the dataset and inference.

• For note types:
  • 1 denotes a rest
  • 2 denotes a regular note
  • 3 denotes a slur
    These behave similarly to a VAD mask.
• If you plan to train your own model on a custom dataset, I recommend removing the note type feature to reduce complexity and potential label noise.
• The project structure follows the FastSpeech2 design. Components are organized by category.
• Meanwihle, I’ve provided:
  • A fast inference pipeline to evaluation and deployment.
  • The GTSinger dataset, which can be used for both training and inference.

[jayzen33]

Hi! Here’s a clearer overview of the repository structure and how to work with it, along with a refined explanation of ep_types and notes on the dataset and inference.

• For note types:

  • 1 denotes a rest
  • 2 denotes a regular note
  • 3 denotes a slur
    These behave similarly to a VAD mask.

• If you plan to train your own model on a custom dataset, I recommend removing the note type feature to reduce complexity and potential label noise.

• The project structure follows the FastSpeech2 design. Components are organized by category.

• Meanwihle, I’ve provided:

  • A fast inference pipeline to evaluation and deployment.
  • The GTSinger dataset, which can be used for both training and inference.

Thank you for your reply, which offers invaluable insights for my own research as well as that of others in the field. It will certainly enhance the visibility and impact of this project.