Default GH runners run out of resources when building custom MFE images for Pearson

[Serafin-dev]

Description

We encountered an issue while building MFE images for Pearson.
Two different builds stopped at approx. 29 to 31 mins of the process, exactly at the 'Build service image with no cache' step.
There are no error logs, only this message from the action:

We think it is a shut down signal triggered due to lack of resources.

[felipemontoya]

@Serafin-dev we'll be investigating this in the next sprint. Could you please let us know which runner you were using for this action?

All the info you can give us would be great.

[Serafin-dev]

@felipemontoya We are building openedx images using Standard Github-hosted runners but this is not working for building MFE images. We have added a larger runner to try again. I am testing it right now, so I will provide feedback asap.

[Serafin-dev]

Hi @felipemontoya. We were able to build mfe images by using this runner:
.
This indicates that it was a resource issue. Anyway, if you find that a larger-runner is not needed, please let me know.

update 12/19/24
This runner worked for the first attempts, but a custom MFE was added recently that made it to run out of memory.

[Serafin-dev]

On 12/18/2024 I performed some tests to monitor system resources while building MFE images and
found that at both cases(Default runners and the Large runner we are using) when building MFE images for our specifc case, both run out of memory.
I did that by monitoring each runner resources during the build process, and analazing and comparing outputs.

Default runner 2-cores 8GB GB SSD

Metric	Start	End
Number of Processes	1 running, 0 blocked	6 running, 31 blocked
Memory Usage	614Mi used, 5.2Gi free, 6.8Gi available	6.1Gi used, 147Mi free, 1.3Gi available
Swap Usage	3.0Gi total, 0B used, 3.0Gi free	3.0Gi total, 2.6Gi used, 374Mi free
Disk Usage	55Gi used, 18Gi free (76% used)	65Gi used, 7.8Gi free (90% used)
CPU Usage	13% user, 4% system, 81% idle, 1% waiting	38% user, 20% system, 19% idle, 22% waiting

Large runner 4-cores 16GB RAM 150GB SSD

Metric	Start	End	Observations
Processes	3 running, 0 blocked	17 running, 28 blocked	Increase in processes.
Memory (free)	13.5 GiB	210 MiB	Memory almost completely consumed.Likely caused by parallel builds or inefficient resource allocation
Disk Usage	52 GiB used, 36% utilization	71 GiB used, 49% utilization	increase in disk usage during the job.
CPU usage	~0%	14% user, 5% system, 81% idle	CPU underutilized despite memory exhaustion.

Large runner 8-cores 32GB RAM 300GB SSD -> This was able to complete the build and the one we will be using.

Metric	Start	End
Memory (free)	30Gi	14Gi

I think it is worth considering further optimization for the Large runner 4-cores case, as it can reduce costs and get the most out of it.
As of now, the max-parallelism parameter is being configured for Docker Buildkit and was tested using '3' as its value.
An observation was that by default, Default runners has swap memory available in case of running out of RAM, but that is not the case for Large runners, leading to a shut down when the limit is reached. However there are actions that can be used for this matter.

As runners duplicate their resources, also does billing -> ref. https://docs.github.com/en/enterprise-cloud@latest/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates-for-x64-powered-larger-runners , so if this is a concern, we should try to select a runner that uses the max of its resources when properly optimized, while not incurring in a system shut down.

[MoisesGSalas]

I will close the issue given that we have two workarounds for this problem:

1. We recently replaced the ENABLE_LIMIT_BUILDKIT_PARALLELISM input with BUILDKIT_MAX_PARALLELISM in feat!: allow specifying the parallelism limit for buildkit #41 to allow more granular control over how many layers can build at the same time. For the most part this should work in most cases unless one single layer is exhausting all resources.
2. Use a different runner via the RUNNER_WORKFLOW_LABEL input (large-runner or self-hosted).

Seems like using the 8-core runner should work on all scenarios. In my tests limiting the max-parallelism to 3 was enough to make the defaults runners work. You may check if different values for the parallelism can net you the sweet spot in the 4-core runner.