Support idle runners > max runners

[SystemKeeper]

I am wondering if the following would be possible somehow:

• Have max runners set to 5
• Have idle runners set to 8
  => 5 runners are actively running workflows, while another 3 runners are already prepared (so the 'installation' step is already done).
  When one of the 5 running workers finishes, it gets deleted and one of the already prepared runners now run the next workflow. A new container is started and prepared.

[gabriel-samfira]

Hi @SystemKeeper !

The problem is that we can't control which runners get picked up by jobs and which do not. It is not garm that allocates jobs to runners. The runners, once installed and joined to github, will pull jobs from Github once they are available. There is no way to "reserve" a runner for later use.

The only way that could potentially happen, is if we set up the runner but we never join it to github. But currently that is not possible, as we don't have an agent running on the runners. We spin them up and set them up via userdata. Once they boot, that user data makes sure they are set up properly and it's out of garm's control.

I'm guessing you want to reduce the amount of time you wait for a new runner to be spun up. If that is the case there are a couple of things you can try:

• make sure the image you're using is up to date. This will shorten the amount of time needed for updates.
• bundle the software you need inside the image you're using. A lot of time may be spent installing tools.
• increase the number of min-idle-runners. This can incur cost, but having more idle runners means that during periods of high demand, you'll have a bunch of them ready to pick up the jobs.

There have been discussions to add some logic behind the min-idle-runners option and make it adapt to demand. But that is something we'll be able to tackle a bit further into the future.

[SystemKeeper]

Hello @gabriel-samfira,

thank you very much for the detailed information! I see that this is not easily possible, I wasn't aware on how the jobs are distributed.

I'm guessing you want to reduce the amount of time you wait for a new runner to be spun up. If that is the case there are a couple of things you can try:

Exactly, this was my idea: Reduce the time for spinning up a new runner and no need to download/install the runner each time (I also saw the work at #84). I am just thinking out loud here, would something like this work:

• Spin up template-container-{IMAGE-HASH}-{RUNNER-HASH}
• Install dependencies and the runner inside the container
• For each runner use that container and instead of installing the runner, just start it
• If a new version if the runner is published (so the hash changes), remove the old template container and create a new one

bundle the software you need inside the image you're using. A lot of time may be spent installing tools.

For the runner itself this is needed, or is there something else that could be prepared? https://github.com/actions/runner/blob/main/docs/start/envlinux.md

[gabriel-samfira]

The PR you linked to, actually adds the ability to bundle the runner inside the image itself. You have two options when doing so:

• Add the extracted runner to /opt/cache/actions-runner/latest in which case, garm won't do any version checking and will blindly trust that whatever you put there is indeed the latest. This is useful if you want to run a pre-release of the runner or have your own patches applied to it.
• Add the extracted runner to /opt/cache/actions-runner/$VERSION where $VERSION is the version of the runner. In this case, if what garm fetches from github is different than what you bundled in the image, it will download and install the version indicated by github.

Note, when bundling the runner with your image, you will have to download it, extract it to one of the above mentioned locations and also run the ./bin/installdependencies.sh inside the extracted folder. All dependencies needed to run the runner must be pre-installed when bundling.

Important note: Garm does not deal with preparing and maintaining templates. It simply consumes whatever image/template you configure when defining the pool. It is out of scope for garm to build images. This needs to be a pipeline you set up yourself, as there may be other tools you may wish to bundle with your images.

For example, you may choose to preinstall compilers, interpreters, libraries, utilities, etc. You will also have to keep the image packages as up to date as possible. This is not something that garm should do as it's a potentially complex pipeline in and of itself.

[SystemKeeper]

Thanks again for the explanation! I did miss the first option about the latest directory, I'll try that.

About managing of images, I can see that this is out of scope of garm. I simply thought preparing an already provided image with the current runner and then use that might make sense. But then again, we could already prepare the image in the needed way.

[gabriel-samfira]

About managing of images, I can see that this is out of scope of garm. I simply thought preparing an already provided image with the current runner and then use that might make sense. But then again, we could already prepare the image in the needed way.

I try to fight the urge (usually works out, sometimes not tat much 😄) to feature creep things into projects I maintain, just because they seem cool to have and/or "should not be that hard to add". In my experience, having focus in a project is really important. You want your project to do the thing it was designed to do and do it really well, instead of doing many things but poorly.

There are so many great tools out there that can help us build CI pipelines that generate proper images that garm could just consume, that it's hardly worth duplicating what other people worked really hard to polish. One such tool is Hashicorp's Packer.

[SystemKeeper]

I try to fight the urge (usually works out, sometimes not tat much 😄) to feature creep things into projects I maintain, just because they seem cool to have and/or "should not be that hard to add"

Oh I can relate to that ;-). All good from my side. I'll try bundling the runner within the image and see how this works.

[gabriel-samfira]

If you create a pipeline for building custom VM images (container images already exist) and feel it's something you can share, I'd be happy to ad a link in our readme to your article 😁 .

[SystemKeeper]

Currently we try to be as close to the GitHub runner images as possible. We use the patch from https://github.com/whywaita/actions-runner-images-lxd to build the original image to be used in LXD. There are still some difference between the images run on GitHub actions and the ones we build for LXD, but it's quite close. But that's still container images and not VM images, so I at the moment, I can't be of help here.

[SystemKeeper]

• Add the extracted runner to /opt/cache/actions-runner/latest in which case, garm won't do any version checking and will blindly trust that whatever you put there is indeed the latest. This is useful if you want to run a pre-release of the runner or have your own patches applied to it.

Btw: I added this to our image and it's working great!

[gabriel-samfira]

Currently we try to be as close to the GitHub runner images as possible. We use the patch from https://github.com/whywaita/actions-runner-images-lxd to build the original image to be used in LXD. There are still some difference between the images run on GitHub actions and the ones we build for LXD, but it's quite close. But that's still container images and not VM images, so I at the moment, I can't be of help here.

This project looks interesting. Thanks for sharing!

[gabriel-samfira]

Closing this issue as some of the aspects described here cannot be helped, and some have been merged.