[Warning] High Utilisation of GitHub Runners

[lupyuen]

Description / Steps to reproduce the issue

Our usage of GitHub Actions Runners has hit 70 Full-Time Runners over the past 12 hours. Remember that ASF Infra Team has demanded that we keep this Under 25 Full-Time Runners, or we might lose access to GitHub Actions CI.

https://infra-reports.apache.org/#ghactions&hours=12&limit=15&group=name

I'm not sure whether this is due to high volume of PRs in NuttX Repo or NuttX Apps. Or possibly some runaway jobs? Let's continue to monitor this, hope it will drop within the next few days.

The URL above is accessible to NuttX PMC Members (possibly to NuttX Maintainers too). Other folks may use this simpler metric for monitoring: https://lupyuen.github.io/nuttx-metrics/github-fulltime-runners.png

I'm travelling overseas for a Marathon Race, my response will be slow. Thanks!

FYI: This is how we redesigned NuttX CI, 13 months ago: https://lupyuen.org/articles/ci3

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu on GitHub Actions

NuttX Version

master

Issue Architecture

[Arch: all]

Issue Area

[Area: Other]

Host information

No response

Verification

• I have verified before submitting the report.

[simbit18]

Thank you so much @lupyuen !

@acassis @cederom @linguini1 @raiden00pl @xiaoxiang781216 @Donny9 @fdcavalcanti @tmedicci @wangchdo @xuxin930 @extinguish @CV-Bowen @zhhyu7 @GUIDINGLI @hujun260 @no1wudi @anchao

I believe it is necessary to slow down/block PR for a few days, otherwise we may lose access to GitHub Actions CI..

#14376

[acassis]

@lupyuen @SiM, is there some way we could create distributed CI buildings to relieve (alleviate) the runners pressure?

[raiden00pl]

Is it possible to temporarily disable PRs in GH? I'm afraid that not everyone follows the GH notifications and mailing list, especially since in recent weeks these has been flooded with changes from Xiaomi

[simbit18]

Hi @xiaoxiang781216, Could you kindly inform the Xiaomi developers of the current situation so that they temporarily suspend the addition of new PRs?

[linguini1]

We're going to need to come up with a more permanent solution again soon.

[acassis]

@simbit18 yes, maybe they can focus on fix the existing PRs that are in development and coordenate between them to only do git push after previos PR finish the CI

[simbit18]

Hi @acassis, the problem is that we have exceeded 25 full-time runners and are now at 54, so we should stop.

[acassis]

@linguini1 seems like we are in a stage similar to what Greg was facing when he are doing "CI" in machine. We thought that after moving to Apache Foundation all our problems will be fixed, this is very disappoint.

I think they should allows to use more than 25 runners, because this is the amount allowed to all projects, even projects that are not active anymore. It should be better to redistribute it between active projects (maybe the situation is a little different than what I'm seeing, but who knows?)

Maybe we will need to run local machines to do the CI and just send the result to github, but since our user base is small I don't know if we will have enough computing power this way

[lupyuen]

Maybe this will help: I see our devs pushing updated commits, which re-runs the build, and wastes GitHub Runners. (Pic below)

Could we ask our devs to wait for the PRs to finish the builds in their own NuttX Repos, before submitting the PR? Remember that the builds in their own NuttX Repos, won't count towards our quota for GitHub Runners. Thanks!

[linguini1]

@linguini1 seems like we are in a stage similar to what Greg was facing where he are doing "CI" in machine. We thought that after moving to Apache Foundation all our problems will be fixed, this is very disappoint.

I think they should allows to use more than 25 runners, because this is the amount allowed to all projects, even projects that are not active anymore. It should be better to redistribute it between active projects (maybe the situation is a little different than what I'm seeing, but who knows?)

Maybe we will need to run local machines to do the CI and just send the result to github, but since our user base is small I don't know if we will have enough computing power this way

I agree it would be nice if resources were redistributed, but I think we can still get a little more clever with our CI. I remember Lup made some really big improvements before. For instance, now we don't run a full build on every documentation change, just a docs rebuild.

I wonder if we can try to do something similar again. Ex: changes to documents under arm/rp2040 only perform builds of rp2040 boards. It's going to be hard to do that for everything but it might be a better option than self hosting?

Also, I wonder if we can cut down on "duplicate" boards in the regular CI. As in, it's not necessary to test a rebuild of every SMT32H7 board when there's a change, just one should be fine in most cases.

[acassis]

@lupyuen is there some way to force Github to post pone (ignore momentarily) the new PR build? I think it will difficult to user wait the right moment to push their PRs.

[acassis]

@linguini1 seems like we are in a stage similar to what Greg was facing where he are doing "CI" in machine. We thought that after moving to Apache Foundation all our problems will be fixed, this is very disappoint.
I think they should allows to use more than 25 runners, because this is the amount allowed to all projects, even projects that are not active anymore. It should be better to redistribute it between active projects (maybe the situation is a little different than what I'm seeing, but who knows?)
Maybe we will need to run local machines to do the CI and just send the result to github, but since our user base is small I don't know if we will have enough computing power this way

I agree it would be nice if resources were redistributed, but I think we can still get a little more clever with our CI. I remember Lup made some really big improvements before. For instance, now we don't run a full build on every documentation change, just a docs rebuild.

I wonder if we can try to do something similar again. Ex: changes to documents under arm/rp2040 only perform builds of rp2040 boards. It's going to be hard to do that for everything but it might be a better option than self hosting?

Also, I wonder if we can cut down on "duplicate" boards in the regular CI. As in, it's not necessary to test a rebuild of every SMT32H7 board when there's a change, just one should be fine in most cases.

Yes, we discussed this idea with @lupyuen in the past: if a PR is only modifying an specific board it doesn't make sense to test all other boards of the same arch. And if only a chip from a specific arch is modified it doesn't make sense to test all ARM boards.

But I think these rules are difficult to implement on our CI. Maybe we need to rethink our CI from scratch