Performance Improvement ideas / feature requests

[UsernamesLame]

As promised, here's the thread I'm making for this.

RE: pre-processing:

In pywhispercpp/model.py we have transcribe and it can take a numpy ndarray. What I was thinking is, rather than load in audio, crush it to mono, set it to 16khz, why not pre-process all that and generate binary blob files that we can feed in that just contain the numpy ndarray?

It's not a big performance increase, but anything we can do outside of Python land ahead of time will give us a win. And I'm ok chasing micro-optimizations in Python land. I'm useless in C++ land.

Also let's put all logging behind a flag to disable it. If possible, lets add a flag to disable whisper.cpp's incessant logging info to stderr. I know it has no impact on the transcription audio, but it should be controllable.

RE: copy.deepcopy

We need to drop @statimethod everywhere, and implement the deep copy methods on the C++ side. This is a minor request from me, it would just let us initialize the model in memory and create a deep copy that we can treat as a completely independent instance.

The other option is I can write a helper class using BytesIO to hold the model in memory and we can feed that to the Model class I guess? It would still be better than re-initializing the model to create a sterile instance.

RE: micro-optimizations

Under _get_segments we have assert end <= n, f"{end} > {n}: `End` index must be less or equal than the total number of segments" but I have to ask, is it even possible to end up in a situation where this assert would come true?

RE: features

Lets make the model usable in a context manager so we can do quick and dirty things like:

with Model("base.en", n_threads=6) as model:
    for segments in model.transcribe("file.mp3")
        for segment in segments:
            print(segment)

Not really necessary, just gives a more pleasant way of interacting with the model class.

[UsernamesLame]

Looked into the numpy array saving: https://numpy.org/doc/stable/reference/generated/numpy.save.html

We can save the converted audio files to disk before feeding them to the model. This way we technically bypass the need for PyDub and ffmpeg. It also means no launching background processes (PyDub with ffmpeg) to manipulate audio so it's ready for the model to ingest.

[absadiki]

@UsernamesLame, Thanks for the ideas!

• I don't think I understand the first point correctly, maybe some code will make it clear.
• About logging: Yes it's annoying that logs are written to stderr, it's possible to add the flag, but needs some tweaks.
• copy.deepcopy: What's that for ? You can create as many instances as you want! Maybe some code will be useful in here as well.
• _get_segments: Yes might happen, if you want to get segments more than what whispercpp actually generated.
• The context: Good feature, I actually started it at that time but I don't remember what happened why it's not there 😅

[UsernamesLame]

This is what I was trying to explain:

sound = AudioSegment.from_file(media_file_path)

sound = sound.set_frame_rate(constants.WHISPER_SAMPLE_RATE).set_channels(1)

arr = np.array(sound.get_array_of_samples()).T.astype(np.float32)
arr /= np.iinfo(samples[0].typecode).max

with open("file.npy", "wb") as file:
    np.save(file, arr, allow_pickle=False)

I haven't tested it yet, but the idea is do all the operations we need on the numpy array ahead of time and then later we can just do something like:

array = np.load("file.npy")

_transcribe(array)

This way we can mass process our audio files before we load them into memory for whisper to process.

@abdeladim-s hopefully this makes sense now! The idea is to batch process hundreds if not thousands of audio files ahead of time in parallel (I can write a script to do this for us) and save them in a format we can just load into the model and get transcriptions back from.

Yes I know numpy should be fast but every context switch we can avoid the better.

[absadiki]

Looked into the numpy array saving: numpy.org/doc/stable/reference/generated/numpy.save.html

We can save the converted audio files to disk before feeding them to the model. This way we technically bypass the need for PyDub and ffmpeg. It also means no launching background processes (PyDub with ffmpeg) to manipulate audio so it's ready for the model to ingest.

PyDub and ffmpeg are actually there for the conversion to numpy arrays!
If we have numpy arrays, why we would need to save them to disk ?

[UsernamesLame]

Looked into the numpy array saving: numpy.org/doc/stable/reference/generated/numpy.save.html
We can save the converted audio files to disk before feeding them to the model. This way we technically bypass the need for PyDub and ffmpeg. It also means no launching background processes (PyDub with ffmpeg) to manipulate audio so it's ready for the model to ingest.

PyDub and ffmpeg are actually there for the conversion to numpy arrays! If we have numpy arrays, why we would need to save them to disk ?

Pre-processing. Every context switch we can avoid the better! Imagine transcribing thousands of files.

The current solution looks like this:

pywhispercpp -> PyDub -> ffmpeg -> PyDub -> pywhispercpp -> numpy -> pywhispercpp -> PyBind11 -> whisper -> PyBind11 -> pywhispercpp

With my proposal it would look more like this:

pywhispercpp -> numpy -> pywhispercpp -> PyBind11 -> whisper -> PyBind11 -> pywhispercpp

The goal isn't to make this a full replacement for the existing solution, but I tomorrow I'll write a demo showing an alternative to load data into the model cutting out as many context switches as possible to gain some performance.

[absadiki]

This is what I was trying to explain:

sound = AudioSegment.from_file(media_file_path)

sound = sound.set_frame_rate(constants.WHISPER_SAMPLE_RATE).set_channels(1)

arr = np.array(sound.get_array_of_samples()).T.astype(np.float32)
arr /= np.iinfo(samples[0].typecode).max

with open("file.npy", "wb") as file:
    np.save(file, arr, allow_pickle=False)

I haven't tested it yet, but the idea is do all the operations we need on the numpy array ahead of time and then later we can just do something like:

array = np.load("file.npy")

_transcribe(array)

This way we can mass process our audio files before we load them into memory for whisper to process.

@abdeladim-s hopefully this makes sense now! The idea is to batch process hundreds if not thousands of audio files ahead of time in parallel (I can write a script to do this for us) and save them in a format we can just load into the model and get transcriptions back from.

Yes I know numpy should be fast but every context switch we can avoid the better.

Okey, so the idea is to process large amount of files ?
But I think it's the same, if not worse, taking into consideration the overhead of saving and loading the files to/from disk. And you will need to wait for the conversion in any ways.
IO operations are worse than using memory.

[UsernamesLame]

This is what I was trying to explain:

sound = AudioSegment.from_file(media_file_path)

sound = sound.set_frame_rate(constants.WHISPER_SAMPLE_RATE).set_channels(1)

arr = np.array(sound.get_array_of_samples()).T.astype(np.float32)
arr /= np.iinfo(samples[0].typecode).max

with open("file.npy", "wb") as file:
    np.save(file, arr, allow_pickle=False)

I haven't tested it yet, but the idea is do all the operations we need on the numpy array ahead of time and then later we can just do something like:

array = np.load("file.npy")

_transcribe(array)

This way we can mass process our audio files before we load them into memory for whisper to process.
@abdeladim-s hopefully this makes sense now! The idea is to batch process hundreds if not thousands of audio files ahead of time in parallel (I can write a script to do this for us) and save them in a format we can just load into the model and get transcriptions back from.
Yes I know numpy should be fast but every context switch we can avoid the better.

Okey, so the idea is to process large amount of files ? But I think it's the same, if not worse, taking into consideration the overhead of saving and loading the files to/from disk. And you will need to wait for the conversion in any ways. IO operations are worse than using memory.

IO operations are generally cheaper than context switches. I'll test this unless you want to.

I can also read the files into memory and store them in a BytesIO object and read from it like a filesystem object too. There's a lot of ways this can be taken. But I genuinely believe that avoiding context switches > IO

RE: deepcopy

You can create completely independent objects that are clones of existing objects. Think instead of myModel = Model, we do myModel = existingModel.deepclone(). So we don't read the model weights from disk again, but instead do an in memory copy.

[absadiki]

IO operations are generally cheaper than context switches. I'll test this unless you want to.

I can also read the files into memory and store them in a BytesIO object and read from it like a filesystem object too. There's a lot of ways this can be taken. But I genuinely believe that avoiding context switches > IO

RE: deepcopy

You can create completely independent objects that are clones of existing objects. Think instead of myModel = Model, we do myModel = existingModel.deepclone(). So we don't read the model weights from disk again, but instead do an in memory copy.

Yes please, go ahead and test! experiments and Numbers will save us a lot of talk :)
Looking forward it!

[UsernamesLame]

IO operations are generally cheaper than context switches. I'll test this unless you want to.
I can also read the files into memory and store them in a BytesIO object and read from it like a filesystem object too. There's a lot of ways this can be taken. But I genuinely believe that avoiding context switches > IO
RE: deepcopy
You can create completely independent objects that are clones of existing objects. Think instead of myModel = Model, we do myModel = existingModel.deepclone(). So we don't read the model weights from disk again, but instead do an in memory copy.

Yes please, go ahead and test! experiments and Numbers will save us a lot of talk :) Looking forward it!

So I'm testing wth a sample 33mb mp3 and the results are promising. Pre-processing into a numpy array and saving to disk shrinks it to 5.4mb so we can definitely have an impact on memory footprint with a helper script! Let me test transcription performance.

[UsernamesLame]

I have numbers for you @abdeladim-s!

Here's the script:

``` py from pywhispercpp.model import Model import numpy as np import time

def usenumpy():
model = Model('base')
audio_data = np.load("file.npy")
segments = model.transcribe(audio_data)
for segment in segments:
print(segment)

def useaudiofile():
model = Model('base')
segments = model.transcribe("audio.mp3")
for segment in segments:
print(segment)

begin = time.time()
usenumpy()
end = time.time()
print("" * 20)
print(f"using raw numpy array finished in {end - begin}")
print("" * 20)

begin = time.time()
useaudiofile()
end = time.time()
print("" * 20)
print(f"using mp3 file inished in {end - begin}")
print("" * 20)

</details>



Here's the results!

---

using raw numpy array finished in 2.6472320556640625

---

---

using mp3 file inished in 26.56456184387207

---

On a M1 Pro MBP with 16gb of ram, **not** using the Metal backend, using the **base** whisper model.

Told ya it would have an improvement on processing time to pre-process the audio files into Numpy arrays!

This computer has a memory bandwidth of 200GB/s, and disk bandwidth of around 4GB/s. Context switching costs more than just loading raw data into memory :)

I am going to chase every optimization I can like a dog chases its tail.

[UsernamesLame]

from pydub import AudioSegment
import numpy as np

sound = AudioSegment.from_file("audio.mp3")

sound = sound.set_frame_rate(1600).set_channels(1)
arr = np.array(sound.get_array_of_samples()).T.astype(np.float32)

with open("file.npy", "wb") as file:
    np.save(file, arr, allow_pickle=False)

This is the pre-conversion script. I'm going to update WhisperWav to output numpy arrays that can be fed directly into the model.

[UsernamesLame]

So I tried pre-converting a few files. Most work, but at random Numpy will completely mangle the conversion to a ndarray and saving leading to UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data with numpy.load.

If anyone has any idea why it's randomly mangling things I'd love help here.

Edit:

Yea I'm at a complete dead end as to why numpy insists on butchering audio files at random when saving. When it works, the speedups are insane. When it doesn't work, the errors are absolutely useless.

Edit 2:

I decided to see if Copilot could help. It suggested:

with open("audio.npy", "rb") as f:
    audio_data = np.fromfile(f, dtype=np.float32)

And so far it seems to be working?

[UsernamesLame]

Ok so final comment for now. A 42m audio file at 101 mb once crushed to mono and audio bitrate set to 1600khz becomes a 17mb~ npy file.

Processing the npy file takes around 10 seconds. Processing the raw wav file takes around 63 seconds.

This doesn't seem like an error or unreasonable. Can someone else please try and reproduce? Are we literally spending that much time prepping the file?!

[absadiki]

I still don't get what you are trying to achieve, but if I understand it correctly, it's basically the same as what I did, except that you are trying to dump and load the npy array, and you've made a deadly bug! lol

Also, when you did the experiment, why you didn't calculate the time needed to convert the files to npy, people are not moving around with dumped npy arrays of their media files 😅

Here is what I think this should be:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from pywhispercpp.model import Model
import numpy as np
import time
from pydub import AudioSegment


def usenumpy():
	# This part from your script should be included as well! ##########
    sound = AudioSegment.from_file("audio.mp3")
    # Here 16Khz not 1600 !!!! That's what you were doing wrong !!! 
    sound = sound.set_frame_rate(16000).set_channels(1)
    arr = np.array(sound.get_array_of_samples()).T.astype(np.float32)
    arr /= np.iinfo(np.int16).max # Normalization is important! otherwise you will get 'utf-8' codec can't decode bytes
    # dump array to npy file
    with open("file.npy", "wb") as file:
        np.save(file, arr, allow_pickle=False)
	#################### 
    # load model
    model = Model('base')
    # load array from npy file
    audio_data = np.load("file.npy")
    segments = model.transcribe(audio_data)
    for segment in segments:
        print(segment)

def useaudiofile():
    model = Model('base')
    segments = model.transcribe("audio.mp3")
    for segment in segments:
        print(segment)

begin = time.time()
usenumpy()
end = time.time()
print("*" * 20)
print(f"using raw numpy array finished in {end - begin}")
print("*" * 20)

begin = time.time()
useaudiofile()
end = time.time()
print("*" * 20)
print(f"using mp3 file finished in {end - begin}")
print("*" * 20)

I used this file from my other project, here are the results:

[2024-08-30 17:34:17,168] {model.py:130} INFO - Transcribing ...
[2024-08-30 17:34:33,929] {model.py:133} INFO - Inference time: 16.761 s
t0=0, t1=424, text=[Music]
t0=424, t1=800, text=What exactly is artificial intelligence?
t0=800, t1=1192, text=We speak of AI when computer systems perform tasks
t0=1192, t1=1448, text=that usually require human intelligence.
t0=1448, t1=1624, text=This includes, for example,
t0=1624, t1=2056, text=recognizing images, making decisions or engaging in dialogue.
t0=2056, t1=2624, text=To do this, the AI systems must be equipped with knowledge and experience.
t0=2624, t1=2824, text=This can be achieved in two ways.
t0=2824, t1=3000, text=[Music]
t0=3000, t1=3280, text=You can program each individual instruction
t0=3280, t1=3544, text=so that the machine solve the task step by step.
t0=3544, t1=3984, text=This is comparable to a cooking recipe or assembly instructions.
t0=3984, t1=4480, text=Alternatively, you can use programs that learn from data themselves.
t0=4480, t1=5032, text=This enables them to detect relevant information, draw conclusions, or make predictions.
t0=5032, t1=5336, text=This is known as machine learning.
t0=5512, t1=5904, text=We all have probably dealt with AI at some point in our lives.
t0=5904, t1=6224, text=When we watch films, listen to music or shop online.
t0=6224, t1=6528, text=AI gives us recommendations about what we might like.
t0=6528, t1=7080, text=AI is capable of converting spoken language into text
t0=7080, t1=7312, text=and translating it into other languages.
t0=7312, t1=8040, text=AI is a central component of robotics.
t0=8040, t1=8288, text=Robots make our everyday lives easier
t0=8288, t1=8488, text=or take on strenuous activities.
t0=8488, t1=8984, text=Self-driving vehicles recognise their environment through AI
t0=8984, t1=9096, text=and can react to it.
t0=9096, t1=9568, text=AI is becoming increasingly important within medicine.
t0=9568, t1=9840, text=It supports doctors when diagnosing diseases.
t0=9840, t1=10696, text=Also, more and more patients use AI-based apps for initial diagnosis.
t0=10696, t1=11264, text=In the educational sector, AI helps to individualise learning activities.
t0=11280, t1=11544, text=For example, on digital learning platforms.
t0=11544, t1=11928, text=AI is becoming increasingly important.
t0=11928, t1=12504, text=Once we understand how AI works, we can better gauge where it can support everyday activities
t0=12504, t1=12688, text=at home and at work.
t0=12688, t1=12896, text=And where we would rather make our own decisions.
t0=12896, t1=13512, text=AI will not replace humans, but it is getting better and better at supporting us.
t0=13512, t1=13840, text=For this, we need an AI-competent society.
t0=13840, t1=14176, text=[MUSIC PLAYING]
t0=14176, t1=14376, text=you
********************
using raw numpy array finished in 17.416718244552612
********************
[2024-08-30 17:34:34,516] {model.py:130} INFO - Transcribing ...
[2024-08-30 17:34:50,128] {model.py:133} INFO - Inference time: 15.612 s
t0=0, t1=424, text=[Music]
t0=424, t1=800, text=What exactly is artificial intelligence?
t0=800, t1=1192, text=We speak of AI when computer systems perform tasks
t0=1192, t1=1448, text=that usually require human intelligence.
t0=1448, t1=1624, text=This includes, for example,
t0=1624, t1=2056, text=recognizing images, making decisions or engaging in dialogue.
t0=2056, t1=2624, text=To do this, the AI systems must be equipped with knowledge and experience.
t0=2624, t1=2824, text=This can be achieved in two ways.
t0=2824, t1=3000, text=[Music]
t0=3000, t1=3280, text=You can program each individual instruction
t0=3280, t1=3544, text=so that the machine solve the task step by step.
t0=3544, t1=3984, text=This is comparable to a cooking recipe or assembly instructions.
t0=3984, t1=4480, text=Alternatively, you can use programs that learn from data themselves.
t0=4480, t1=5032, text=This enables them to detect relevant information, draw conclusions, or make predictions.
t0=5032, t1=5336, text=This is known as machine learning.
t0=5512, t1=5904, text=We all have probably dealt with AI at some point in our lives.
t0=5904, t1=6224, text=When we watch films, listen to music or shop online.
t0=6224, t1=6528, text=AI gives us recommendations about what we might like.
t0=6528, t1=7080, text=AI is capable of converting spoken language into text
t0=7080, t1=7312, text=and translating it into other languages.
t0=7312, t1=8040, text=AI is a central component of robotics.
t0=8040, t1=8288, text=Robots make our everyday lives easier
t0=8288, t1=8488, text=or take on strenuous activities.
t0=8488, t1=8984, text=Self-driving vehicles recognise their environment through AI
t0=8984, t1=9096, text=and can react to it.
t0=9096, t1=9568, text=AI is becoming increasingly important within medicine.
t0=9568, t1=9840, text=It supports doctors when diagnosing diseases.
t0=9840, t1=10696, text=Also, more and more patients use AI-based apps for initial diagnosis.
t0=10696, t1=11264, text=In the educational sector, AI helps to individualise learning activities.
t0=11280, t1=11544, text=For example, on digital learning platforms.
t0=11544, t1=11928, text=AI is becoming increasingly important.
t0=11928, t1=12504, text=Once we understand how AI works, we can better gauge where it can support everyday activities
t0=12504, t1=12688, text=at home and at work.
t0=12688, t1=12896, text=And where we would rather make our own decisions.
t0=12896, t1=13512, text=AI will not replace humans, but it is getting better and better at supporting us.
t0=13512, t1=13840, text=For this, we need an AI-competent society.
t0=13840, t1=14176, text=[MUSIC PLAYING]
t0=14176, t1=14376, text=you
********************
using mp3 file finished in 16.196656465530396
********************

This is not a real experiment per say, but as you can see, they are almost the same. There is no need to dump and load the numpy array!

Lmk what do you think ?

[UsernamesLame]

I still don't get what you are trying to achieve, but if I understand it correctly, it's basically the same as what I did, except that you are trying to dump and load the npy array, and you've made a deadly bug! lol

Also, when you did the experiment, why you didn't calculate the time needed to convert the files to npy, people are not moving around with dumped npy arrays of their media files 😅

Here is what I think this should be:

#!/usr/bin/env python

# -*- coding: utf-8 -*-



from pywhispercpp.model import Model

import numpy as np

import time

from pydub import AudioSegment





def usenumpy():

	# This part from your script should be included as well! ##########

    sound = AudioSegment.from_file("audio.mp3")

    # Here 16Khz not 1600 !!!! That's what you were doing wrong !!! 

    sound = sound.set_frame_rate(16000).set_channels(1)

    arr = np.array(sound.get_array_of_samples()).T.astype(np.float32)

    arr /= np.iinfo(np.int16).max # Normalization is important! otherwise you will get 'utf-8' codec can't decode bytes

    # dump array to npy file

    with open("file.npy", "wb") as file:

        np.save(file, arr, allow_pickle=False)

	#################### 

    # load model

    model = Model('base')

    # load array from npy file

    audio_data = np.load("file.npy")

    segments = model.transcribe(audio_data)

    for segment in segments:

        print(segment)



def useaudiofile():

    model = Model('base')

    segments = model.transcribe("audio.mp3")

    for segment in segments:

        print(segment)



begin = time.time()

usenumpy()

end = time.time()

print("*" * 20)

print(f"using raw numpy array finished in {end - begin}")

print("*" * 20)



begin = time.time()

useaudiofile()

end = time.time()

print("*" * 20)

print(f"using mp3 file finished in {end - begin}")

print("*" * 20)

I used this file from my other project, here are the results:

[2024-08-30 17:34:17,168] {model.py:130} INFO - Transcribing ...

[2024-08-30 17:34:33,929] {model.py:133} INFO - Inference time: 16.761 s

t0=0, t1=424, text=[Music]

t0=424, t1=800, text=What exactly is artificial intelligence?

t0=800, t1=1192, text=We speak of AI when computer systems perform tasks

t0=1192, t1=1448, text=that usually require human intelligence.

t0=1448, t1=1624, text=This includes, for example,

t0=1624, t1=2056, text=recognizing images, making decisions or engaging in dialogue.

t0=2056, t1=2624, text=To do this, the AI systems must be equipped with knowledge and experience.

t0=2624, t1=2824, text=This can be achieved in two ways.

t0=2824, t1=3000, text=[Music]

t0=3000, t1=3280, text=You can program each individual instruction

t0=3280, t1=3544, text=so that the machine solve the task step by step.

t0=3544, t1=3984, text=This is comparable to a cooking recipe or assembly instructions.

t0=3984, t1=4480, text=Alternatively, you can use programs that learn from data themselves.

t0=4480, t1=5032, text=This enables them to detect relevant information, draw conclusions, or make predictions.

t0=5032, t1=5336, text=This is known as machine learning.

t0=5512, t1=5904, text=We all have probably dealt with AI at some point in our lives.

t0=5904, t1=6224, text=When we watch films, listen to music or shop online.

t0=6224, t1=6528, text=AI gives us recommendations about what we might like.

t0=6528, t1=7080, text=AI is capable of converting spoken language into text

t0=7080, t1=7312, text=and translating it into other languages.

t0=7312, t1=8040, text=AI is a central component of robotics.

t0=8040, t1=8288, text=Robots make our everyday lives easier

t0=8288, t1=8488, text=or take on strenuous activities.

t0=8488, t1=8984, text=Self-driving vehicles recognise their environment through AI

t0=8984, t1=9096, text=and can react to it.

t0=9096, t1=9568, text=AI is becoming increasingly important within medicine.

t0=9568, t1=9840, text=It supports doctors when diagnosing diseases.

t0=9840, t1=10696, text=Also, more and more patients use AI-based apps for initial diagnosis.

t0=10696, t1=11264, text=In the educational sector, AI helps to individualise learning activities.

t0=11280, t1=11544, text=For example, on digital learning platforms.

t0=11544, t1=11928, text=AI is becoming increasingly important.

t0=11928, t1=12504, text=Once we understand how AI works, we can better gauge where it can support everyday activities

t0=12504, t1=12688, text=at home and at work.

t0=12688, t1=12896, text=And where we would rather make our own decisions.

t0=12896, t1=13512, text=AI will not replace humans, but it is getting better and better at supporting us.

t0=13512, t1=13840, text=For this, we need an AI-competent society.

t0=13840, t1=14176, text=[MUSIC PLAYING]

t0=14176, t1=14376, text=you

********************

using raw numpy array finished in 17.416718244552612

********************

[2024-08-30 17:34:34,516] {model.py:130} INFO - Transcribing ...

[2024-08-30 17:34:50,128] {model.py:133} INFO - Inference time: 15.612 s

t0=0, t1=424, text=[Music]

t0=424, t1=800, text=What exactly is artificial intelligence?

t0=800, t1=1192, text=We speak of AI when computer systems perform tasks

t0=1192, t1=1448, text=that usually require human intelligence.

t0=1448, t1=1624, text=This includes, for example,

t0=1624, t1=2056, text=recognizing images, making decisions or engaging in dialogue.

t0=2056, t1=2624, text=To do this, the AI systems must be equipped with knowledge and experience.

t0=2624, t1=2824, text=This can be achieved in two ways.

t0=2824, t1=3000, text=[Music]

t0=3000, t1=3280, text=You can program each individual instruction

t0=3280, t1=3544, text=so that the machine solve the task step by step.

t0=3544, t1=3984, text=This is comparable to a cooking recipe or assembly instructions.

t0=3984, t1=4480, text=Alternatively, you can use programs that learn from data themselves.

t0=4480, t1=5032, text=This enables them to detect relevant information, draw conclusions, or make predictions.

t0=5032, t1=5336, text=This is known as machine learning.

t0=5512, t1=5904, text=We all have probably dealt with AI at some point in our lives.

t0=5904, t1=6224, text=When we watch films, listen to music or shop online.

t0=6224, t1=6528, text=AI gives us recommendations about what we might like.

t0=6528, t1=7080, text=AI is capable of converting spoken language into text

t0=7080, t1=7312, text=and translating it into other languages.

t0=7312, t1=8040, text=AI is a central component of robotics.

t0=8040, t1=8288, text=Robots make our everyday lives easier

t0=8288, t1=8488, text=or take on strenuous activities.

t0=8488, t1=8984, text=Self-driving vehicles recognise their environment through AI

t0=8984, t1=9096, text=and can react to it.

t0=9096, t1=9568, text=AI is becoming increasingly important within medicine.

t0=9568, t1=9840, text=It supports doctors when diagnosing diseases.

t0=9840, t1=10696, text=Also, more and more patients use AI-based apps for initial diagnosis.

t0=10696, t1=11264, text=In the educational sector, AI helps to individualise learning activities.

t0=11280, t1=11544, text=For example, on digital learning platforms.

t0=11544, t1=11928, text=AI is becoming increasingly important.

t0=11928, t1=12504, text=Once we understand how AI works, we can better gauge where it can support everyday activities

t0=12504, t1=12688, text=at home and at work.

t0=12688, t1=12896, text=And where we would rather make our own decisions.

t0=12896, t1=13512, text=AI will not replace humans, but it is getting better and better at supporting us.

t0=13512, t1=13840, text=For this, we need an AI-competent society.

t0=13840, t1=14176, text=[MUSIC PLAYING]

t0=14176, t1=14376, text=you

********************

using mp3 file finished in 16.196656465530396

********************

This is not a real experiment per say, but as you can see, they are almost the same. There is no need to dump and load the numpy array!

Lmk what do you think ?

I caught the deadly bug locally and fixed it locally.

As for performance, it's odd you aren't getting better results and I am.

I'm guessing it has something to due with the memory bandwidth of M1 Pro vs x86 chips?

But yea you're understanding now. I haven't tested it on x86. Also I didn't include the converting to numpy arrays because the idea is to mass transform it then transcribe.

At least one benefit is the numpy arrays are generally smaller in my experience.

What are your system specs btw? And Python version? I'm using 3.12 and getting good results.

If I can't increase performance I can at least lower memory usage I guess. 😅

My idea is to let the model be long lived and keep feeding it fresh areas dumps as it transcribes them one after another. This way in a different process (I'm going to edit e action to show this) we can spawn sub processes to mass convert media files to numpy arrays.

The idea is that the model is the limiting factor, as in most people don't have the CPU / RAM to load 2 - 4 models, so if we can pre-process the files so the model can transcribe faster with less memory, it's still a (small) win!

I have access to a 128core ARM box that is piss slow at transcribing but can quickly spit out these bumpy arrays.

It's not gonna benefit everyone, but it's worth exploring the thought. It's also possible to store all the numpy arrays in a single database that clients running the models pull from to transcribe creating transcription cluster. The big benefit being that the clients can be small like a raspberry pi and still considerably faster transcriptions.