GPU Utilization

[Ivan2getdmodelz]

I have Ubuntu 22.04 and 24.04 with nvidia GPUs, and noticed when monitoring the GPU, that each time a file is loaded, the GPU gets loaded, and once the transcription is complete, it removes the model from VRAM. For speed and for batch processing (i.e. monitoring the watch folder) a useful feature would be to leave the model in VRAM until either a new or different model is selected, or the program is closed.

[raivisdejus]

We could implement a flag to keep the model in memory. Loading of the model will not give much for large file transcriptions, but for a lot of smaller files there could be a benefit.

If you could describe your use case in a bit more details, maybe that gives some additional ideas for enhancements

[Ivan2getdmodelz]

Understood that for large files, there might not be a significant gain, but for a case where multiple short files are being processed in succession, or batch processed from a watch folder, a large portion of the time per file is taken loading the model into the VRAM. It seems for efficiency sake, leaving the selected model in VRAM, and clearing it only on selecting a differnt model or closing the program seems like a good approach.

[raivisdejus]

@Ivan2getdmodelz Curious of your use case. Can you share what are those short files from or about and how you add them to the Buzz. Maybe we have some hidden user need that needs entirely new solution.