Resulting .srt file loses synch with audio file used for transcription (Google Colab notebook)

[BlohoJo]

Hi! 👋😀

I'm having a problem where the audio I uploaded for Whisper-Diarization loses sync with the srt subtitle file that it generates.

Here's the audio file I'm working with. It's a 16 bit WAV file.

https://drive.google.com/file/d/1aUvTx7dZcfQy7ZdlRTULPEQrl8WHmXyO/

I have left all options in the Google Colab notebook at their defaults (other than the audio file name).

The resulting SRT file loses synch for me at line 31, and continues to get worse from there. You can observe this in Subtitle Edit. (Open the generated .srt file and drag the audio file onto the waveform.)

(Not sure if this is the right place to report this?)

Just for reference, my methodology is:

• Upload WAV file to Google Drive -> MyDrive.

• Open URL https://colab.research.google.com/github/MahmoudAshraf97/whisper-diarization/blob/main/Whisper_Transcription_%2B_NeMo_Diarization.ipynb

• Insert below cell above cell under "Options" to connect to Google Drive:

from google.colab import drive
drive.mount('/content/drive')

• Change Runtime Type, Select High-RAM, Save

• Connect

• Run "Installing Dependencies". When prompted to restart, hit "Cancel" and wait for running tasks to complete before selecting "Runtime" -> "Restart Session".

• Run "Installing Dependencies" again after runtime has restarted.

• Run "Helper Functions"

• Run inserted cell in "Options" to connect to Google Drive.

• Drag the uploaded audio file from "MyDrive" to "sample_data" then drag again to the content folder (up one level from "sample_data"). (Note: this resolves issues with dragging directly from MyDrive to the content folder.)

• Change "audio_path =" under "# Name of the audio file" to WAV audio file name

• Run "Options" cell

• Run "Processing"

Note: The Google Drive upload is to work around issues with upload speed, trunication, and corruption when using the "Upload" function in the Google Colab notebook.

[BlohoJo]

I do see some recent changes under ctc_forced_aligner which may be where the problem is?

[BlohoJo]

Audio file info:

Audio
Count                                    : 280
Count of stream of this kind             : 1
Kind of stream                           : Audio
Stream identifier                        : 0
Format                                   : PCM
Commercial name                          : PCM
Format settings                          : Little / Signed
Format settings, Endianness              : Little
Format settings, Sign                    : Signed
Codec ID                                 : 1
Codec ID/Url                             : http://www.microsoft.com/windows/
Duration                                 : 5513984
Duration                                 : 1 h 31 min
Duration                                 : 1 h 31 min 53 s 984 ms
Duration                                 : 01:31:53.984
Bit rate mode                            : CBR
Bit rate mode                            : Constant
Bit rate                                 : 1536000
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2
Channel(s)                               : 2 channels
Sampling rate                            : 48000
Sampling rate                            : 48.0 kHz
Samples count                            : 264671232
Bit depth                                : 16
Bit depth                                : 16 bits
Stream size                              : 1058684928
Stream size                              : 1 010 MiB (100%)
Stream size                              : 1 009.6 MiB
Proportion of this stream                : 1.00000

Report created by                        : MediaInfoLib - v21.03
Report created on                        : UTC 2025-01-22 15:17:50

[BlohoJo]

I did try the setting whisper_model_name = "large-v3", setting batch_size = 0, and enable_stemming = False. None of them had any impact on the issue. (I tried each setting separately.)

[sourman]

yeah I noticed stride is off by some small factor but the error adds up over time. try adding 1 or 2 to stride in generate_emissions() at the end before it returns and see it might fix the issue

[BlohoJo]

It's not always just off, it often keeps the sub on too long, or not long enough.

I've checked things with timings and sync's to see if it's something simple like a sample rate issue or something like that, but it's not a simple speed change. It starts too soon, too late, too long, or not long enough in seemingly variable amounts at variable times, and it all adds up to be several minutes off by the end.

I've used this notebook in the past with success, I had a nearly 3 hour long podcast I used it with and it stayed in near perfect sync until the end, so, something must have changed.

try adding 1 or 2 to stride in generate_emissions() at the end before it returns and see it might fix the issue

Sorry, but I have no idea how to do that! 😕

[BlohoJo]

Is this what you meant?

Original:

audio_waveform = (
    torch.from_numpy(audio_waveform)
    .to(alignment_model.dtype)
    .to(alignment_model.device)
)

emissions, stride = generate_emissions(
    alignment_model, audio_waveform, batch_size=batch_size
)

del alignment_model
torch.cuda.empty_cache()

Change to (add stride += 1 as shown below):

audio_waveform = (
    torch.from_numpy(audio_waveform)
    .to(alignment_model.dtype)
    .to(alignment_model.device)
)

emissions, stride = generate_emissions(
    alignment_model, audio_waveform, batch_size=batch_size
)

stride += 1

del alignment_model
torch.cuda.empty_cache()

The resulting srt file is worse and a lot more out of alignment than before. Unless you meant something different?

[sourman]

yeah that is what I meant. Hmm has to be something else. can you run git bisect between the version that was working and the current version and single out which commit causes the issue? It might require you to run the application 3-10 times max before the bad commit is identified

[BlohoJo]

Ugh, I don't know how to code Python or anything for that matter, so I've no idea how to use bisect. I used ChatGPT o1 for my previous reply. 😞

I checked and the last confirmed working .srt file I have is dated 10/04/2024. That makes the commit for ctc-forced-aligner that worked for that file https://github.com/MahmoudAshraf97/ctc-forced-aligner/tree/a8d18c399e49b6339198dbbc4cef77468a3aeeed (09/28/2024).

I changed line 5 in cell 1 under Installing Dependencies to:

!pip install git+https://github.com/MahmoudAshraf97/ctc-forced-aligner.git@a8d18c399e49b6339198dbbc4cef77468a3aeeed

It now runs out of memory unless the runtime type is changed to High-RAM.

I ran it with High-RAM, and it worked! It's now in sync, up until the end. However, it doesn't identify individual speakers as every line starts with Speaker 0: .

So, there does seem to be some regression in ctc-forced-aligner at some point.

I'll see if I can figure out bisect, or, if I can't, I can just manually try commits from the above until I hit the regression and post back.

[BlohoJo]

OK, I think I did this correctly:

!git clone https://github.com/MahmoudAshraf97/ctc-forced-aligner.git
  Cloning into 'ctc-forced-aligner'...
  remote: Enumerating objects: 318, done.
  remote: Counting objects: 100% (138/138), done.
  remote: Compressing objects: 100% (40/40), done.
  remote: Total 318 (delta 122), reused 98 (delta 98), pack-reused 180 (from 1)
  Receiving objects: 100% (318/318), 87.72 KiB | 2.92 MiB/s, done.
  Resolving deltas: 100% (184/184), done.

%cd ctc-forced-aligner
!git branch
  [Errno 2] No such file or directory: 'ctc-forced-aligner'
  /content/ctc-forced-aligner
  * main
  
!git bisect start HEAD a8d18c399e49b6339198dbbc4cef77468a3aeeed
  Bisecting: 7 revisions left to test after this (roughly 3 steps)
  [cb27067f735dbac246dc4398f776a56b80b96d78] fix batching condition

!git bisect good
  Bisecting: 3 revisions left to test after this (roughly 2 steps)
  [2a27d0200d76956ad419b967cb430aa25103f409] fix CI badge [skip ci]
  
!git bisect bad
  Bisecting: 1 revision left to test after this (roughly 1 step)
  [058dc2be8157704575d6c0dbe729be56c5115161] handle MPS device in emissions

!git bisect bad
  Bisecting: 0 revisions left to test after this (roughly 0 steps)
  [134666ea7208fdc2a66e27b3b9518126048bc6cb] formatting [skip ci]

!git bisect bad
  134666ea7208fdc2a66e27b3b9518126048bc6cb is the first bad commit
  commit 134666ea7208fdc2a66e27b3b9518126048bc6cb
  Author: MahmoudAshraf97 <hassouna97.ma@gmail.com>
  Date:   Tue Jan 7 18:16:55 2025 +0200

      formatting [skip ci]

   ctc_forced_aligner/align.py           |  7 ++++---
   ctc_forced_aligner/alignment_utils.py |  2 ++
   ctc_forced_aligner/text_utils.py      | 11 ++++++-----
   3 files changed, 12 insertions(+), 8 deletions(-)

[sourman]

Excellent work. git bisect saves so much time. now that we know the offending commit debugging will be much easier. I will take a look at it this weekend.. In the meantime keep using the first git commit. in this case the one right before 134666ea7208fdc2a66e27b3b9518126048bc6cb according to your logs

[MahmoudAshraf97]

Thanks for your efforts @sourman , I checked 134666ea7208fdc2a66e27b3b9518126048bc6cb and the following commits but all what was changed is CI related and Readme, do you have any findings so far?

[sourman]

Happy to help :) no I have not looked at the diffs. Strange. CI updates wouldn't break this 🤔