Could the whisper training set / model architecture help with this project?

[johndpope]

At first glance
https://github.com/openai/whisper
it seems unrelated.
I’m interested in how stable diffusion and audio could help make timbre transfer easier.
Seems like breaking audio into words and aligning this could be part of the puzzle to effectively do timbre transfer.
Does this project intend to incorporate stable diffusion down the track? Or is that not relevant/ applicable?

[flavioschneider]

What do you mean with "incorporate stable diffusion"? Stable diffusion is an architecture to generate images. If you mean text conditioning, that is already possible by providing text embeddings during training and sampling.

Whisper is for speech recognition and this repo is for generation, those are not directly compatible as long as you don't have something more specific in mind.

[johndpope]

Thanks for quick response Flavio.
I came across some work by @hmartelb who took a melspectrogram image and used pix2pix to attempt timbre-transfer here
https://github.com/hmartelb/Pix2Pix-Timbre-Transfer
I raised a ticket in that repo - but given this repo is proclaiming to pytorch audio diffusion + generation of audio - I've added here too.

Just thinking out loud - but consider that the whisper framework (speech to text)
could be hotwired to instead of detection of text - it could also match to a phoneme -
https://github.com/openai/whisper/blob/main/whisper/model.py#L162
or maybe like given a phoneme -> generate to a target (timbre transferred) phoneme.

I found this - which is using diffusion
https://github.com/MoonInTheRiver/DiffSinger

it seems like a big part of why these models suck so bad for tts is they don't have so much training data.

[johndpope]

fyi https://github.com/teticio/audio-diffusion