Tests of arm64 based images are not performed

[consideRatio]

Assuming we merge #1399, then we only run tests for the amd64 based image.

My suggestion is that we try to acquire arm64 based self-hosted GitHub runners that can assist this project and other projects in the Jupyter ecosystem. If we have that, then we can run tests and such easily.

[manics]

I've been looking at free resources for self-hosted GitHub runners. The most promising looks like Oracle Cloud which has "4 Arm-based Ampere A1 cores and 24 GB of memory usable as one VM or up to 4 VMs" in the "Always Free" service

• https://www.oracle.com/uk/cloud/free/#always-free

You need a credit card to sign-up to prove your identity (and presumably to stop you signing up for multiple free accounts), but they say they won't use it if you go over the free resource quota unless you switch to a paid account:

• https://www.oracle.com/uk/cloud/free/faq.html

GitHub have a "Deploy to Oracle Cloud" button for Arm to make it easy to setup:

• https://www.google.com/search?client=firefox-b-d&q=oracle+github+runner+

There are disadvantages though:

• You're using the same VM every time, not a freshly provisioned one, so you have to be careful of what you globally install or configure
• Untrusted code will be run on the VM if it's used for testing PRs. I think you can use job and runner labels to control which job runs where, but you'd need to be very careful to avoid a security hole

[trallard]

Following here and since I already mentioned it in #1407 (comment)

We can ask for permanent free access to machines at https://github.com/WorksOnArm/equinix-metal-arm64-cluster - this could help us unblock arm-related build/test

I can help with submitting the proposal - but also know @mathbunnyru offered to help

[mathbunnyru]

@trallard

I can help with submitting the proposal - but also know @mathbunnyru offered to help

Please, proceed and ask for arm runner, I don't have much spare time right now.
I will be glad to help if I can ❤️

Thank you for your ideas and working on this project 👍

[consideRatio]

If we have that, then we can run tests and such easily.

I've learned a few things that are problematic. I suggest we go for it anyhow in this case, but I just wanted to provide a heads up about two points.

Challenges with self-hosted arm runners

1. The ARM runners can't use github actions like actions/setup-python or actions/setup-node etc, because they rely on pre-installed Python/Node versions in the runner. They also make checksum checks etc which makes this not possible to workaround by providing such installations for arm.
2. The self-hosted runners can be provisioned in different ways, and only by having fresh VMs has felt secure enough. I attempted to provide self-hosted runners on raspberry pi computers starting up containers in a k8s cluster but abandoned the idea after a while as I didn't feel confident enough with regards to security.

[consideRatio]

Scope of request

Do we scope the request for arm runners to be this project, jupyter github org, or jupyter ecosystem as a whole etc?

I suggest we scope it for this project as a pilot, and then later with experience we can consider asking for further help for runners etc.

If we have it functional well here, I'd appreciate these runners in jupyterhub/zero-to-jupyterhub-k8s as well for example, but I don't want to bog down the process with creeping the scope - whats important now is that we pilot this to function well enough I think.

[trallard]

Thanks for the insights @consideRatio

I have little experience with self-hosted arm runners but being aware of these limitations and risks early on is very helpful.

Maybe we can add these or some other concerns to the request form? I am not sure if the folks providing these for OSS projects have also some sort of docs on how to secure the runners but worth asking.

Also I believe these are granted on a per project basis - so we would start with this project and treat it as a "pilot " like you mentioned. And if it works well proceed with other projects.

I will try and get the ball rolling but I also anticipate I will need help with some information gathering (as I might be missing some historical context) and perhaps @consideRatio can help with a review before I submit anything

[consideRatio]

@trallard I'd be happy to help with review!

We are approved - then what?

I would love if we had a practical example to follow from someone else, where someone has already used this free-infra offering to host self-hosted github actions runners. I worry that without it getting the machines will be the easy part, while setting up everything beyond that to help us have self-hosted runners in the way we wish is the hard part.

I think it will be far more reasonable to have self-hosted runners that are running code only when we merge to master etc - in other words - running code as approved by maintainers. It is more problematic if we are to run the self-hosted runners as part of running tests for PRs.

Related discussions

• https://github.com/WorksOnArm/equinix-metal-arm64-cluster/issues/210
• https://github.com/WorksOnArm/equinix-metal-arm64-cluster/issues/303#issuecomment-1041511827

Then we setup a persistent VM to run the github self-hosted runner software?

I'm not sure what the followup would be if we get this approved. But, I'm thinking that we would setup a few GitHub Runners to run on the same VM, and that those would be used only for trusted workflows such as merging to the default branch or scheduled executions etc - not for any incoming PRs.

I'm not confident if you can run multiple versions of the self-hosted runners on the same machine, but I've seen indications that this can work.

Capacity to follow through?

Making the request is step 1, after it is approved we need to do more work no matter if its a relatively simple plan to just have a fixed VM running the github actions self-hosted runners.

Is there anyone willing to put in work with regards to this? I'm not having much time available to offer =/

[trallard]

I should have some capacity to follow through.
Might need to do loads of research first but I can certainly dedicate some time to this