Port build reproducibility and security

[Lgt2x]

Hi there! First of all, I hope this is the right place to discuss this. I've recently been looking into Linux handheld gaming software, and PortMaster seems to be a project on the rise when it comes to running desktop games.

Running PortMaster for the first time was a pretty smooth experience, but looking into it more left me with a few questions that are not addressed in the FAQ. Adding a new port looks very straightforward: a README, a few xml/json metadata files, and a bash script doing whatever it needs to get the game to run.

What was more surprising to me was seing raw binary executable embedded in the repository (making it really large and cumbersome to clone without LFS :/ ), especially for games derived from open source engines. I don't feel confident running binaries added by the community through Pull Requests that anyone can open, for 2 main reasons, the first one being security. Does PortMaster provide any sand-boxing or analysis for games? If it does not, some bad actor could push compromised binaries without anyone knowing; a compromised device could leak sensitive data (SSH keys, WiFi Access Point passwords...). Even then, handhelds usually have internet access, giving potential attackers full access to LAN, which is something you usually want to avoid.

Additionally, as the number of ports is increasing steadily, this also raises the question of maintainability and reproducible builds. Let's say I want to update the engine for a recently added port, or troubleshoot a game not booting on a given device (by the way, how do we know which games can run or have been tested on a given device?). Looking at a random port https://github.com/PortsMaster/PortMaster-New/blob/main/ports/descent/README.md , I could not find any instruction to build the game or even source location, which basically means that the effort made to get the game to compile has been lost.

Is there anything in place or planned to mitigate these issues? Even requiring a BUILD.md, Dockerfile and hash for binaries could help a lot to get more accessible and reproducible builds.

Thanks anyway for running this project improving the Linux handheld experience!

[JeodC]

@Cebion and @kloptops can get into more detail, but I can raise a few points to help out.

Port Security

Ports are tested in our Discord server before we merge them. If a pull request doesn't have an accompanied Testing Thread with passing marks, it sits in limbo or gets denied. The same goes for port updates, excluding minor updates like metadata changes. As we're a relatively niche community, we rely on users to report oddities. We have yet to have a security breach, fortunately.

Maintainability & Reproducability

PortMaster is only a few years old, and our standardizations are still changing. There are a lot of old ports that don't measure up to current times, including logging and as you pointed out, build steps. In my case, I try to keep my own ports maintained separately before I submit pull requests to this repository.

Ship of Harkinian is one of the more recent ports that is consistently updated and maintained, since it's one of the most popular.

Dockerfile and hash

We do this stuff as a hobby, so if we lack experience for something that should probably be done to make things easier, it generally doesn't get done.

I hope you're enjoying your handheld! Don't be shy in the Discord. :) Plenty of friendly folks.

[Lgt2x]

Ports are tested in our Discord server before we merge them. If a pull request doesn't have an accompanied Testing Thread with passing marks, it sits in limbo or gets denied. The same goes for port updates, excluding minor updates like metadata changes. As we're a relatively niche community, we rely on users to report oddities. We have yet to have a security breach, fortunately.

Manual testing cannot ensure that the game has not been tampered with or the lack of a backdoor. Truly open sourcing ports and documenting the build process is the only way to get trust-worthy binaries shipped. The real problem here is that handheld users may download executables that PortMaster maintainers do not have the source for, and did not build themselves.

We do this stuff as a hobby, so if we lack experience for something that should probably be done to make things easier, it generally doesn't get done.

This is also why I'm opening this issue! This problem seems very critical to me for the project in the long run. Raw binaries in a git repository is not manageable at scale, and I'm all in to help improve standards for the ports hosted here.

I hope you're enjoying your handheld! Don't be shy in the Discord. :) Plenty of friendly folks.

As much as I like Discord for certain things, messages tend to get easily lost and buried quickly, so I turned in here to try and get more public attention

[JeodC]

I mean, it is true that we currently have no methods for verifying, say, libraries bundled with a port. A backdoor would be relatively easy to implement. Again, fortunately, it hasn't been a problem thus far, and, as the community grows, more will take notice and the probability of it increases, especially with recent developments like Valve and Arch Linux.

Regarding open source ports, I'm sure you've seen that not every port we have is OSS. Many, like Descent 3, are just compatibility layers without the game assets.

In terms of solution, we have no solution (yet). It hasn't been a high priority for us. It appears you're working to understand the entire structure of the project in hopes of proposing one or more solutions though, which we do appreciate.

[Cebion]

Dear @Lgt2x ,

thank you for your thoughtful feedback and concerns regarding PortMaster. We appreciate you taking the time to look into our project in depth. Let me address your main points:

Security and Binary Executables:
We understand your concerns about running binaries from community contributions. While we don't currently have an automated sandboxing or analysis system, we do have a manual review process:

• All ports are tested by our community before merging.
• Pull requests require an accompanying Testing Thread with passing marks.
• Our relatively niche community helps report any oddities.

However, we acknowledge this doesn't fully address the potential for malicious code. We're open to improving our security measures and would welcome specific suggestions or contributions in this area.

Maintainability and Reproducible Builds:
You're correct that some of our older ports lack comprehensive build instructions. We're actively working to improve our standards. For newer ports, we aim to include:

• A README with compilation instructions
• Source code changes when modifications are made to open-source projects
• Build steps in the port documentation

For example, here's a snippet from one of our READMEs with compilation instructions:
https://github.com/PortsMaster/PortMaster-New/blob/main/ports/kuklomenos/README.md

## Notes
Thanks to [Martin Bays](https://mbays.freeshell.org/kuklomenos/) for creating this awesome game.

## Controls

| Button | Action |
|--|--| 
|DPAD| Move|
|ESC| Menu|
|Start| Space (Start Game)|
|A| Gun 1|
|B| Gun 2|
|Y| Gun 3|

## Compile ## 

```bash
git clone https://github.com/Cebion/Kuklomenos.git
cd Kuklomenos
./configure
# adjust prefix to . 
make

It's worth noting that these concerns have been on our radar for some time.
About two years ago, one of our contributors, pkegg, initiated work on a PortMaster-Build system (https://github.com/Cebion/PortMaster-Build) that aimed to address some of these issues.

While that specific implementation wasn't fully realized at the time, it demonstrates our community's ongoing efforts to improve our build processes and address concerns like the ones you've raised.

As our project has grown from around 40 ports on the old PortMaster Project to over 635 on the PortMaster-New Project , we've continually evolved our practices. The recent migration to our new GitHub organization marks another step in this direction, allowing us to implement more robust processes and tools. Imagine that on the old repo every port was uploaded as a zip file where we could not track the changes at all. (https://github.com/christianhaitian/PortMaster) We're always looking for ways to improve, and your feedback helps us identify areas where we can focus our efforts.

Repository Size and Cloning:
We understand the issue with repository size due to binary files. We're exploring options to improve this, possibly including better use of Git LFS or alternative distribution methods for larger files.

Future Plans:
Your suggestions for improvement are valuable. We're considering:

• Requiring BUILD.md files for new ports
• Implementing a more structured approach to documenting build processes
• Exploring options for reproducible builds

Open Source vs. Proprietary Games:
It's worth noting that not all our ports are open-source. Some, like gmloader games, are compatibility layers for proprietary games. This limits our ability to provide full source code in some cases.

We're a community-driven project, and all of us work on this as a hobby. While this sometimes means we lack resources for implementing every best practice, we're always striving to improve. Our project's history demonstrates this commitment to growth and adaptation.

We welcome your continued input and would be thrilled if you'd like to contribute to improving these aspects of PortMaster.

If you have specific ideas or would like to help implement any of these improvements, please let us know.

Thank you again for your interest in making PortMaster better and more secure for everyone.
Your feedback is an important part of our ongoing evolution as a project.
Best regards,

Daniel

[Lgt2x]

Thank you for your quick and thorough reply. I took some time to think about constructive suggestions.

About security, I don't think there is a silver bullet, shipping binaries from the community is risky no matter how you do it. Runtime security with sandboxing could be achieved using lightweight containers like lxc, making sure games don't have access to the whole host filesystem. This may cause be some difficulties in practice regarding save management using that, but it could be worked out.

A good and easier way to reduce the attack surface would be factorisation of the dynamic libraries included in every game, for instance providing a global library of trusted .so files copied to the games folders when creating the release. It would also help reduce the repository size. This could also be done for game engine runtimes such as Godot or Löve.

As for source builds, compilation steps as a bash script are great, but the gold standard is still a Dockerfile, that also manages build dependencies and is independent of the host platform. Requiring a Dockerfile for open source games distribution would be much more valuable for the transition towards more reproducible builds than simple instructions that are often platform-dependent.

The PortMaster-Build project looks very good, it could be worthwhile to invest more resources into it if you can. Do you remember why it did not get more community adoption? If it's a question of complexity for less technical contributors, it could be great to slap a GUI on top of it to manage builds, find game files if necessary and easily transfer games through SSH to the target hardware for testing. If that's of any interest, I can try and get a proof of concept going.

I agree that git LFS is quickly going to become an absolute necessity. However as you probably know, GitHub does not provide it for free, and you pay for the bandwidth. If PortMaster already has a dedicated server, I would recommend hosting git LFS (or any FTP/SFTP solution) there instead to reduce costs.

It's worth noting that not all our ports are open-source. Some, like gmloader games, are compatibility layers for proprietary games. This limits our ability to provide full source code in some cases.

Sure, but even for proprietary games, the binary blobs should have their source listed, and a method provided for anyone to retrieve it independently.

We're a community-driven project, and all of us work on this as a hobby. While this sometimes means we lack resources for implementing every best practice, we're always striving to improve. Our project's history demonstrates this commitment to growth and adaptation.

Again, I really appreciate your feedback and your serious consideration of the issues raised here. Building this kind of project in public is very valuable for the community as a whole, and I hope we can collectively elevate it to the next level, both technically and in terms of public adoption.

[Cebion]

Thank you again for your thoughtful suggestions and engagement with our project. After discussion with our team, we'd like to provide a more detailed response that addresses your points and clarifies our current situation and future plans.

We appreciate your suggestions regarding Docker and standardized build processes. We do indeed have Ubuntu 20.04 Docker images available for those who wish to use them. However, we've intentionally maintained flexibility in how contributors compile their binaries. Our community uses a diverse range of methods, including:

• Cross-compilation
• WSL2 Chroots
• Raspberry Pis for native ARM compilation
• Various other setups

This flexibility has been crucial to our project's growth and accessibility. While we see the value in standardization, we're cautious about implementing strict requirements that might deter potential contributors.

We agree that the PortMaster-Build project was conceptualized with the right intentions. Back then there were about 2-3 contributors in total and no one showed real interest. Our experience has shown that maintaining a low barrier to entry is vital for attracting and retaining contributors. The current popularity of our project is largely due to this accessibility.

Implementing a system similar to Buildroot's package descriptions would be a significant undertaking and could potentially discourage casual contributors. While we appreciate the benefits of such a system (particularly the ease of updating once set up), we have to balance these against the risk of losing contributors.
That said, we see potential in applying this approach to our core tools and runtimes like gptokeyb, gmloader, runtimes, and others.

For these critical components, a more structured build system could be beneficial. Your suggestion of a GUI to manage this process is intriguing and could indeed make it more accessible.

Git LFS:
You're correct that we haven't yet reached the limit of our releases. However, we're aware that this will become an issue in the future as our project continues to grow. We're actively monitoring our usage and will explore alternatives when necessary. This might include self-hosting solutions to manage costs effectively.

It's crucial to understand that PortMaster is a relatively small project. Our core members, who drive the project forward, do so entirely in their spare time. One of our biggest challenges is supporting a wide range of ecosystems.
We're not working with a single Linux OS or set of drivers, instead, we deal with a variety of environments, for example:

• ArkOS with its full Ubuntu CFWs (Custom Firmwares) with Ubuntu 19, which use older glibc versions
• Devices running Tina Linux with even more limited features such as TSP
• CFWs without busybox like muOS
• Systems with read-only partitions like AmberELEC and Rocknix

This diversity is why suggestions like "just use Flatpaks" aren't feasible for us. Our project needs to be adaptable to these various environments, each with its own limitations and quirks.

It's important to note how far we've come. PortMaster started as a one-person project with a simple dialog frontend and about 30 ports, lacking any build documentation. We've made significant strides in standardization and documentation since then. We must remain realistic about what's achievable given our resources and the need to maintain an accessible environment for contributors. We need to balance this with maintaining our contributor base

In theory implementing a complex, multi-tier hardening system might provide an excellent security, but it would be counterproductive if it resulted in a loss of most of our contributors. In short, the project would be dead.

So moving forward we will start by picking up the low hanging fruits

• Continuing to improve our build documentation
• Exploring ways to consolidate and vet common libraries
• Enhancing our processes for core tools and runtimes
• Maintaining and improving our documentation of binary sources

Most binary blobs we use for runtimes are currently documented under https://portmaster.games/porting.html

If you're interested in contributing directly to any of these efforts, particularly in areas like improving our build processes for core tools or enhancing our documentation, we'd be the last to say no.

If you want we can create development channels in our discord just for this cause and will try to get the Crew behind this effort.

[Lgt2x]

This flexibility has been crucial to our project's growth and accessibility

It's a great way to introduce more contributors to the project, but in the end it's a trade-off with long-term maintainability. Many successful software projects constraint the possible tool chains for this reason. That was also why I was suggesting a UI running the container and commands for you, so the build system is abstracted, and contributors can focus on the proper porting work, allowing quicker iterations in a standardized environment.

We're not working with a single Linux OS or set of drivers, instead, we deal with a variety of environments

Correct me if I'm wrong, but for all of these distributions, a single ARM build is provided? Standardizing the build process instead of pushing bare binaries totally goes in that direction of flexibility over the target. One could even imagine easily running PortMaster on the RISC-V devices that will inevitably come around one day.

If you're interested in contributing directly to any of these efforts, particularly in areas like improving our build processes for core tools or enhancing our documentation, we'd be the last to say no. If you want we can create development channels in our discord just for this cause and will try to get the Crew behind this effort.

Sure, build systems is something I'm really keen on, I could be willing to assist with that and try to push the community in a more sustainable direction. I cannot promise how much time I'd be able to put into that, but I can at least try.