Extract and delete mod .zip folders when installing

[afontenot]

I am planning on implementing this myself

No

Describe your request

My Mods directory is about double the size it should (intuitively) be, because the Cache/ contains virtually every large file from my installed mods (it's about 98% audio files). Wouldn't it be much more efficient to just unpack all the files from all the mods to separate directories - one per mod - and not have a separate "cache"?

Additional context

I attempted to prevent the audio banks from being extracted to disk by changing the advanced option UnpackFMODBanks to false, but this crashed with SIGABRT after a second or two on the main menu of the Strawberry Jam modpack. Not totally sure what that option does or if it's currently supported.

[Kalobi]

There are already known issues with too many open files during loading. Having every single mod unzipped does not sound feasible, especially for Mac/Linux.
The crash from UnpackFMODBanks does sound like it should be investigated though. Could you post your log.txt from such a session?

[afontenot]

There are already known issues with too many open files during loading. Having every single mod unzipped does not sound feasible, especially for Mac/Linux.

Seems like it could be an advanced option, maybe? As you say, too many open files is already an issue. In fact I got a crash when trying to load Strawberry Jam because my open file limit was only 1024. I had to increase it to load the modpack. Given this this (presumably) isn't considered a bug with Strawberry Jam or Everest, increasing the number of files by a further fraction doesn't seem too unreasonable to me, given the (quite significant!) disk space savings that would result from this change. (I don't know much about the Everest internals, but maybe it's possible to read more assets into memory at load time so that Celeste doesn't need to hold open a file descriptor?)

The crash from UnpackFMODBanks does sound like it should be investigated though. Could you post your log.txt from such a session?

I didn't want to report this because I'm on Linux and build Everest myself. Not exactly a supported configuration. Interestingly, with this crash, my log.txt doesn't contain anything after "DONE LOADING". However the terminal output captures the crash, and I've attached the result below.

Looks like the relevant bit is probably

Assertion at ../../../mono/mini/mini-exceptions.c:2401, condition `(guint8*)&sa >= (guint8*)tls->stack_ovf_guard_base + tls->stack_ovf_guard_size' not met

Not sure if the tls here is Transport Layer Security or not - if so it seems kind of weird to be caused by the UnpackFMODBanks option, but it's entirely reproducible for me.

I could potentially do a debug build of Everest if that might be helpful. I'm running a release build of the latest stable (with custom patches to get correct version numbering).

log.txt

[Kalobi]

Do you get the same issue with the release build of Everest? Either way, the manual build shouldn't have this issue either.

[afontenot]

Do you get the same issue with the release build of Everest? Either way, the manual build shouldn't have this issue either.

Yes I get the exact same error with an official build.

[afontenot]

Wouldn't it be much more efficient to just unpack all the files from all the mods to separate directories - one per mod - and not have a separate "cache"?

Follow up to this - I've discovered that you can basically do it manually. Everest will accept either ModName.zip or an extracted ModName/ directory. Even better, Everest automatically uses the unzipped directory for audio banks - it doesn't try to cache it. It does cache the DLLs (which probably counts as a bug), but in my case that's only ~10 MB or so.

The big downsides to this are (a) this doesn't happen automatically, and more importantly (b) this disables mod updating inside Everest. The latter makes it unusable for most people. Fortunately I've written an extensive amount of tooling around Everest already, so I'll just write a mod updater myself that does everything I want. (Anyone else encountering this issue might want to check out mons, but it's missing features I need.)

[coloursofnoise]

Mod loading is based on the ID specified in the everest.yaml file, so it does not matter what the folder or zip is named.

Unzipped DLLs being cached is intentional, as they have to be relinked against the appropriate framework and any other dependencies, and sometimes patched by Everest and/or MonoMod.
I wasn't aware that Audio files were also cached and I'm not sure the reasoning behind it, but if it's causing significant issues then it should definitely be looked into.

You cannot update mods in folders, as it is assumed that if the mod is unzipped then you are developing it, and so updating automatically could cause unintended data loss.

There are also much more frequent issues with open file limits when mods are extracted, which has been significant issue with Safari since it can automatically extract files for you.
Texture loading is currently multithreaded in Everest, which does lead to open file limits as well and is a known bug.

Lastly, can you elaborate on the features you feel are missing in mons? If you are already patching or creating additional tools it would be beneficial to other users to have them included upstream.
Building Everest from source is definitely supported, there are currently no modifications made to the CI/release builds that I'm aware of apart from injecting the version number.

[coloursofnoise]

Regarding audio caching, I found this commit that seems rather relevant 😅

Add FMOD bank data caching (possibly too wasteful)

[afontenot]

@coloursofnoise Thanks for the insight!

if it's causing significant issues then it should definitely be looked into.

I guess it's only an issue in the sense that it greatly increases the disk space required to install and use mods. E.g. with a clean Everest install, Strawberry Jam takes almost double the size (2.5 GB) when you use the zip version of it and its dependencies. Obviously, since assets aren't really compressible, zipping them would save negligible space even if the biggest ones weren't being cached.

These days, 2.5 GB is basically nothing, but I have a pretty small SSD, so the difference matters to me.

Building Everest from source is definitely supported

That's good to know. In the past the situation wasn't so great, When I started doing this I had to download metadata from the build pipeline because releases weren't getting reliably uploaded to Github. Still, the home page doesn't even mention that building from source is possible.

Lastly, can you elaborate on the features you feel are missing in mons?

Rather than clog up this issue, I filed an issue on mons itself for you to look at: coloursofnoise/mons#4

[Kalobi]

Building from source isn't mentioned on the website because there's no reason to do it unless you're developing Everest. It's described in detail in the README on Github.

[afontenot]

Since much of this discussion is about the crash, would it be helpful to split that report into a new issue?