Continuous testing for FreeBSD

[asomers]

I'm working on a buildbot cluster that could support continuous testing on FreeBSD for several projects. I've got a prototype running, and you can see a PR in action at asomers/mio-aio#2 . The hardest question is how to secure it. Since anybody can open a PR, that means that anybody can run arbitrary code on the buildslaves. The potential damage is limited; each project gets its own worker, each worker runs in its own jail, and there's a timeout on each build. I could use the firewall to prevent workers from sending email and stuff, but I can't completely isolate workers from the internet without breaking a lot of builds. There are a few options to improve the security situation.

1. Don't automatically build a PR until a maintainer posts a specific comment. This is what open-zfs does. It completely eliminates unreviewed code from running on the workers. However, it's inconvenient for people who are accustomed to Travis building stuff without needing to be asked.
2. Destroy and reclone the worker for each build. This is what Travis does. The worker still runs untrusted code, but not for long. The untrusted code can't modify the filesystem in any persistent way.
3. A hybrid of the previous two. Do a build when a maintainer give the magic comment, but also do builds automatically whenever a PR is posted by a known-good contributor. The list of contributors would probably have to be maintained by hand, but it could be much larger than the list of maintainers.

Does anybody have any better ideas?

[notriddle]

Since nix uses bors, your cluster doesn't need to have its own support for magic comments. All it needs is to be able to build branches on nix's repository and report results using the GitHub Commit Status API. The builds will run if a reviewer calls bors try or bors r+.

On the other hand, if you want to run all the pull requests, and you want suggestions to be better able to isolate the builds, you could use a project-owner-managed whitelist of internet hosts. That would be about as close as you could possibly get to eliminating all risk of bad actors using your build cluster to send comment spam.

[asomers]

Interesting. How would bors request a build from buildbot?

[notriddle]

It drops a merge commit into the right branch, either "trying" for a try build or "staging" for a merge build. BuildBot needs to have a webhook set up so that it automatically builds when those branches change.

[asomers]

Great news! It's easy to setup that kind of hook, as long as the branch name matches a regex. I'll try it out tonight.

[Susurrus]

I wanted to point out here that GitLab CI already supports FreeBSD with their runners. We'd need to have this repo on GitLab to utilize this, but I wanted to point this out in case it's a reasonable path forward. If they've already solved the hard part and it just requires installing a runner, then that seems pretty low-effort. I do not, however, know their plans for future platform support moving forward, but it is a FOSS project, so there's that benefit.

[notriddle]

BuildBot supports FreeBSD fine, as well. The question is how it should be configured.

[asomers]

@Susurrus that's nice. I didn't know GitLab had its own CI system. And it's hosted version looks free, which is nice. Price will be an issue, though, since you need someplace to host the runners. Gitlab's free shared runners are all Linux, and it looks like they don't even have container-level isolation. I'll have to do some more research to see how hard it would be to share a FreeBSD runner (with container-level isolation) amongst multiple projects.

One big downside to switching to GitLab would be losing test coverage on OSX. While there are OSX VPS services, their pricing is unattractive, especially without containers.
https://www.hostmyapple.com/macvps.html

Thanks for the tip. It looks like I've got some more homework to do.

[asomers]

I got a FreeBSD runner setup to work with a GitLab project, and successfully completed a build. Unfortunately, GitLab doesn't yet have any support for sharing FreeBSD runners between projects in a containerized way. My next task will be to see how hard it would be to add that feature.
https://gitlab.com/asomers/mio-aio/commit/0323a4ab6a6e299d5930fb7ae628a6a23c90d8cb/pipelines

[Susurrus]

Your GitLab link is broken. Is that repo public there?

[asomers]

Whoops! Try again.

[Susurrus]

I can see the project, but nothing in it. No files, commits, or pipelines are public.

[asomers]

I'm clearly a beginner at Gitlab. Try again. I think you should be able to see everything and submit pull requests, but not trigger CI builds.