Dev update 6/27 - Database work

[Geektoolkit]

I want to do a quick post about where things are at. The projects gaining some eyeballs which is great, and there are some good energetic discussions around Exif, performance, large file lists, automatic on/off timers, ubuntu and even windows builds. All good stuff.

What's the latest in the development branches
At about 2.18 or so I had a stable build. I decided to go after a stretch goal...blurboxing, and while deep in the image code I decided it'd be a good time to re-write the image display into a control This cleaned up a ton of code, and after fighting with Skia for a while I was able to get a blurbox successfully implemented. However it took a deep performance hit, as images were getting rendered twice...once in the foreground, and then a second image was getting resized into the background.

Since I was deep in rewrite territory, I decided to test out Exif data...the good news is that once it was loaded it was really great, but the terrible news is that to parse it for say 500 images took...well a long time.. Even if they didn't have EXIF data. This trashed load/start times, and also took a huge hit on memory consumption.

At that point I had to make a call...try to fix the perf issue, throw out Exif data, or investigate a path forward. Qwiksilver, myself, and a few discussions had talked about the idea of moving to a database. I didn't want to have a heavy server hosted database, so I investigated sqllite, which is essentially a file that stores an entire DB inside of it with quite a few amazing features. .netCore and Entity Frameworks supported sqllite directly, so I had all the tools I needed. I decided to add that.

This bring us to where we are now. I have a development branch that has sqllite working. Load times still are terrible, but once they're loaded the performance feels pretty nice. The best part is instead of storing every single file's full path in memory (about 70 bytes per entry based on file path lengths and file names) we now just store the database index (as an int, so 4 bytes per entry). Also exif data, file type (image/video) and more are all stored in the database, so the 4 bytes gets us access to all of that data pretty quickly and efficiently.

A back of the napkin calculation for 500 items shows the memory usage for the playlist was reduced by 95% before even taking on the exif data. Also the exif file loading went from 22 seconds to 6ms once the data is in the database (we still take that 22 second hit on startup currently). This means we get more data, faster. And the database file is 32kb for 500 entries currently.

So that's where I am. I'm excited to finish this work as it's going to open up a ton of opportunities...for instance while only jpg supports exif, we could use the database to store data for png files if desired. Also it makes it easy to extend the information in the future for things like proxy files if we have an unsupported format. And Exif data alone allows for great flexability on sorting and filtering of images, taking playlists well beyond just a folder structure (though I'm keeping that as well!)

So then what?
Once that database is locked in, I'm going to work on rewriting the infobar (think full flexability for what data and where it goes..for instance datetime and author in the upper right for instance, or just the title in the lower center). The other work that needs to be done is the sync feature needs to be rewritten. This would use time syncs so that all frames would change at a certain time (would require the clocks to be synced, but NTP / internet connected raspbian OS's do this by default if you set the time zones.
And then of course Youtube videos to show usage, troubelshooting, and basically document it a bit better. I see a lot of questions that really could be helped if I just demo it in a new, fresh video end to end.

So that's where things are at. Thanks for the patience while I work through this, but know that I read every post, every issue, every discsussion, and I want to keep improving and bringing this forward. Thankyou everyone for your support, passion, and kind words.