Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove/Cleanup binary blobs from the repository #720

Open
Wartori54 opened this issue Jan 7, 2024 · 0 comments
Open

Remove/Cleanup binary blobs from the repository #720

Wartori54 opened this issue Jan 7, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@Wartori54
Copy link
Contributor

I am planning on implementing this myself

Yes

Describe your request

In conjunction with @AtomHare

Currently, the Everest repository has lots of binaries (in the form of libraries required in runtime or executables required for building). This is a really efficient practice for getting it up and running, but in the long run, it's a really bad practice.
The current approach has the following issues and/or concerns:

  • There are currently multiple version mismatches with libraries between platforms and bundles. (it clearly can be seen a lot of random versions and basically not matches)
  • From a cybersecurity perspective, it raises considerable concerns. There is no verification for the binaries that contributors upload, and it is not hard at all to upload compromised binaries.

Therefore, we suggest eliminating all of the binary blobs present in the repo and moving them to another system where they will be built from source. (The implementation details will be expanded on later.)

Consequently, this change would:

  • Make the repository truly open source to what's achievable.
  • Improve the management of library versions.
  • Ability to track down what version a user is running from its log, no more exceptions on blank lines, and consistency in debugging symbols paths.
  • Increased security, binaries are guaranteed to not be manipulated.
  • Cleanness, there's not a bunch of binaries all around anymore, and, more importantly, all binaries are in the right place and there's no extra ones. (See https://github.com/EverestAPI/Everest/blob/dev/lib-ext/lib64-osx/libtheorafile.dll, a Windows dll in macOS libs.)
  • And in consequence of the above, better organization and proper directory names with documentation of what each one specifically does. (See lib64-win-x86)
  • Everest has been accumulating technical debt over the years, so this change would lessen that (although not by a lot).

Finally, moving on to the implementation details.

Two options have been considered:

  • Creating a script that sets up all the libraries (downloaded from official sources).
  • Creating another repository that is in charge of holding references to the other projects (as submodules), and via CI, it gets built so that it just has to be pulled on the Everest CI and squashed together.
    Honestly, the second option looks most promising. But we are open to suggestions and feedback.

Additional context

Unfortunately, there are some binaries that are not trivial to obtain, neither from source code nor from official sources. They are listed below:

  • Celeste.exe (no legal sources)
  • fmod dlls (requires an account and web scraping)
  • steam_api64.dll (requires an account and web scraping)

Furthermore, when having to decide which commit or tag to use for building the sources, we'll need Everest's team's help since there's no recurring version for either vanilla or Everest libraries.
Finally, for some libraries specific to MacOS devices, it has not been possible to track down sources for them. Those are listed below:

  • libMoltenVK.dylib
  • libvulkan.1.dylib

Thank you for your time, and please seriously consider the implementation of this upgrade.

@Wartori54 Wartori54 added the enhancement New feature or request label Jan 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant