Could torchcrepe be used instead of the Tensorflow one?

[alexirae]

Hi @rakuri255!

I was wondering if you have considered replacing the current crepe implementation (Tensorflow) with a PyTorch one (Torchcrepe)

Thx!

[RobUmf]

It was suggest.
#249
To my understanding it is still hard coded to use to CREPE
But...

[rakuri255]

@alexirae I agree, we should switch to a different pitching model. This would allow us to upgrade the Python version. We’ve used Torchcrepe before (and in PR #249), but the results were unfortunately worse.
I think we just dont use Torchcrepe correctly

[alexirae]

Thanks @RobUmf / @rakuri255 for the info!

Maybe some potential options could be SwiftF0 or FCPE

From the Overall Results of Pitch Benchmarks SwiftF0 seems to be a little bit better than CREPE (would need to be tested with UltrastarSinger).

On the other hand, the Torchcrepe discrepancy you saw in the test, could be due to what is described in the Decoding section of Torchcrepe, chances are that it needs to be set with:

# Decode using weighted argmax (as in the original implementation)
torchcrepe.predict(..., decoder=torchcrepe.decode.weighted_argmax)

To match the original CREPE implementation and hopefully get a similar score.

My 2 cents!

[RobUmf]

I think I have 4 to see if I can get PR.
I'm really doing this to have A.I do what "I" want.
How I got myself in this rabbit hole
story telling. . good Think make a good D&D master. . . . code little issues . . graphics spotty and inconsistent style . . . debate, looks promising.
Well add that as different PR Think there four modification to consider.

1. don't separate done need to create PR
2. use the snapshot, cache
3. bypass Musicbrain and use text file
4. add torchcrepe Think it is done have not tested it.

and for the overmodulation of the mp3 files in current evaluation.
ruff draft I have better way to clean up vocals
also my switchboard and monitor works but hard links and in .sh
Not forgetting the batch optimizer of existing karaoke songs. reports are done the py are segmented right now.
Easy one but in sh Batch of the music library for portability. Don't need the big MP3 360 or flac files 128k good for most headsets and portable speakers

Hopefully we can get someone who really knows what they are doing.
think rakuri255 is busy with life.

I'm just adding this for reference: >whisper_large-v3 . .JSON to text
robumf@robumf-HP-EliteDesk:/media/robumf/1.5nvme61/UltraSinger-main$ python3 -c "import json; data = json.load(open('/media/robumf/1.5nvme61/AI_Work/Outputs/Bucketeer/Unknown Artist - vocals (2)/cache/whisper_large-v3_cpu_None_16_float32_None.json')); print('\n'.join([s['text'] for s in data['segments']]))"

Spectrograph example
ffmpeg -i "/media/robumf/1.5nvme61/AI_Work/derpy/instrumental.mp3" -lavfi showspectrumpic=s=1280x720:legend=1 spectrogram.png

[rakuri255]

@alexirae thanks for the input! There are finally some competition to crepe. We should add/try these.
@RobUmf yes, currently life hits. Didnt had time to get again fully into it.

[rakuri255]

I currently trying to fix the installer and after some sleepless nights i managed to move to uv.
crepe broke with these old dependencies the whole installer. Also it keeped us because of Tensorflow to python 3.10.

@alexirae thanks to you i droped crepe with SwiftF0 and we now on python 3.12 and can upgrade some packages 🤣

Will close this discussion when it get merged.

[alexirae]

hey @rakuri255!!! those are massive good news! thanks a lot for taking the time to get into that, much appreciated! 👏

[rakuri255]

https://github.com/rakuri255/UltraSinger/tree/fix-installer-with-uv-and-swiftF0-pitcher

[magicus]

How's the pitching with swiftF0 compared to crepe? Did you evaluate? Or was it more "crepe has issues, and also TensorFlow, let's try switching to swiftF0 and take it from there?"?

[rakuri255]

It was more. Fu*k Tensorflow+cuda.
In my "2 Songs" test, it is the same, but faster.

[magicus]

I wonder if it would be worth trying to revive this patch, or at least the idea of using torchcrepe, as an alternative...

Otoh, when I tested on some of the songs I am trying to convert right now, I got better results than I ever gotten with crepe. So maybe it's just better to stick with swiftF0.

[rakuri255]

You can't really compare them. I've already made several tonal adjustments to reduce the number of '~' symbols and fixed invalid notes by key. So with crepe it was also good then. For a true comparison, you would need to disable these changes directly in the code.
But when i compare the amount of the key fixings, swiftF0 seems nearly the same.
Also possible that most of the issues came before the pitching, eg. in the vocal separation or noise filtering.

By the way, the changes have been merged into the master branch

[magicus]

I was playing around with crepe_notes, running it on the same separated vocal file as UltraSinger used, and consistently got a vastly better result than US, as determined by comparing the midi output. (Turns out there is a surprisingly good midi-comparison plugin for VS Code!)

So while there certainly can be issues in vocal separation or noise filtering, it was also obvious that it is possible to do a better job in converting the separated vocal track into midi than what US managed. This was with version 0.0.13-dev8.

I am still intrigued by the method used in crepe_notes, by using the inverse of the confidence score to determine note onset. It would be interesting in trying to implement something like that using swiftF0 as well, given that it too outputs a confidence score.