Error in the "large-v3" model

[chahmatist]

I am writing to bring to your attention an error I encountered while using the "large" model. The error message is as follows:

openai-whisper==20231106

ERROR:root:Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead
Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead

This error appears to be related to an inconsistency between the model's expected input shape and the actual input provided. It suggests that the model expected 128 channels in the input data but received only 80.

I would appreciate it if you could investigate this issue and provide guidance on how to resolve it. If there are any specific steps or workarounds I should follow

[zerochocobo]

_, probs = model.detect_language(mel)

File "/root/miniconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/whisper/decoding.py", line 52, in detect_language
mel = model.encoder(mel)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/whisper/model.py", line 162, in forward
x = F.gelu(self.conv1(x))
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 302, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/root/miniconda3/lib/python3.8/site-packages/whisper/model.py", line 48, in _conv_forward
return super()._conv_forward(
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 298, in _conv_forward
return F.conv1d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead

[Hanysix]

Hello,
you can specify the number of channels in log_mel_spectrogram with n_mels.
mel = whisper.log_mel_spectrogram(audio=audio, n_mels=128)

then you use
_, probs = model.detect_language(mel)

[masao1211]

Hello.
This command can help to solve this issue.
pip install -U openai-whisper
I had the same issue with openai-whisper-20230918.
I solved issue by installing openai-whisper-20231117.

[Pavantejapenugonda]

I see the same issue with the latest version : 20240927

[EdMillard]

I see the same issue with the latest version : 20240927

OpenAI's transcribe code currently gets the number of mel channels from the model and passes it to log_mel_spectrogram() to prevent these number of mel channel mismatches:

mel = whisper.log_mel_spectrogram(audio, model.dims.n_mels).to(model.device)

Someone should probably update the decode() example on OpenAI's whisper home page with this change so people stop tripping over this error.

[liviuiacob]

I had the same problem, but it seems to happen because of a mismatch between the versions.
Maybe you have whisper installed in 2 different locations (ex: one in root, and another one in a virtual environment). You can check where is the used source in your code using:

import whisper
print(whisper.__file__)

And then go to that location and check the timestamp of the files. If they belong to a previous version (before 20231117), it will not work and you will have to update the package in that location, using the following command:

pip install -U openai-whisper

The 2 models (large-v2 and large-v3) have different input shapes: V2 has n_mels=80 and V3 has n_mels=128.

[EdMillard]

OpenAI's transcribe code currently gets the number of mel channels from the model and passes it to log_mel_spectrogram() to prevent these number of mel channel mismatches:

mel = whisper.log_mel_spectrogram(audio, model.dims.n_mels).to(model.device)

Someone should probably update the decode() example on OpenAI's whisper home page with this change so people stop tripping over this error.

[vsuryacharan]

yes there should be a change in home page

[pipoupiwam]

Worked for me, Thanks !

[Elbe94]

Can you help Den 30. okt. 2024 kl. 14.53 skrev vsuryacharan ***@***.***>: yes there should be a change in home page —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[kevin82222]

It's worked, many thanks!