refactor: offload thumbnail rendering to multiple processes

[TheBobBobs]

Summary

Renders can now be queued using the ThumbnailManager class which keeps a pool of processes to perform render jobs.

CacheManager has been rewritten. The caching folder layout has changed to {hash[:2]}/{hash}-{mtime}.webp
It does remove outdated cache items but doesn't support a max cache size.

TODO

• Check that processes start and shutdown properly on all platforms
• Ensure rendering logic produces same results as current implementation
• Expand cache implementation?
• Cleanup code

Tasks Completed

• Platforms Tested:

  • Windows x86
  • Windows ARM
  • macOS x86
  • macOS ARM
  • Linux x86
  • Linux ARM

• Tested For:

  • Basic functionality
  • PyInstaller executable

[VasigaranAndAngel]

Cool!

Which module will handle loading the thumbnails? I want the thumbnail renderer with multiprocessing to be modular so i can generate a specific thumbnail when i wanted for different purposes. for example, i want to generate audio thumbnail as in the following video to use it in the music player widget.

20240930-1809-39.6288321.mp4

[TheBobBobs]

Create an instance of RenderJob and connect it's signals to your code like here.
You can then use the ThumbnailManager instance on Library to queue a render passing in RenderJob.on_finish as the callback function.

[CyanVoxel]

I'm a bit out of the loop on the purpose of this refactor, as the thumbnail renderer was already multi-threaded. I'm also not sure why the CacheManager was rewritten and no longer supports a maximum cache size

[TheBobBobs]

With multi-threading the render threads and main thread have to share the GIL. The multi-thread approach is able to use more cpu still since the libraries used for image manipulation can do work without needing to aqcuire the GIL. But any python code or library code interacting with python objects will freeze the main ui thread.
The cache was rewritten to better work with multiple processes in mind. It is currently very basic but clearing old thumbnails when above max_cache_size shoud be easy to add.

Video showing the ui freeze when rendering thumbnails
Multi-process left Multi-thread right
https://github.com/user-attachments/assets/26fac00a-7dd3-48f4-ba8a-303bd7842ab5