Improve performance of CI system

[consideRatio]

We have crippled our CI systems performance after introducing support for arm64 based images. A key reason for this is that emulation of arm64 images from the amd64 based runners github provide is far worse, besides the fact that we end up building base-notebook and minimal-notebook for arm64 in sequence alongside the other images now.

I'm not fully sure how we should optimize this long run, but under the assumption that we will have high performance self-hosted arm64 based GitHub Action runners that can work in parallel to the amd64 runners. Below is an overview of a very optimized system, where several parts can be done separately.

1. Nightly builds
   We have nightly builds with :nightly-amd64 and nightly-arm64 tags

2. amd64 / arm64 in parallel
   All tests for amd64 and arm64 run in parallel, relying on nightly-amd64 and nightly-arm64 caches

3. Images in parallel where possible
   All tests for individual images are run in a dedicated job that needs its base image job to complete.

   Some images can run in parallel:

   • base
   • minimal
   • scipy | r
   • tensorflow | datascience | pyspark
   • all-spark

4. Avoid rebuilds when merging
   Tests finish by updating a github container registry associated with a PR. By doing so, our publishing job on merge to master can opt to use the images as they were built during tests if they are considered fresh enough.

5. Parallel manifest creation
   Merge to default branch triggers manifest creation jobs on both amd64 and arm64. If we opt to not optimize using step 4 then this could also build fresh images using nightly cache first.

6. Combine manifests into one before pushing to official registry
   Merge to default branch triggers a job that pulls both the amd64 image and arm64 image and defines a combined docker manifest which is then pushed to our official container registry. I think this could be done with something like docker manifest create <name of combined image> <amd64 only image> <arm64 only image> but @manics knows more and I lack experience with this.

Standalone performance issue

This standalone issue will go away by using better strategies like above. It isn't so critical to fix either I'd say. But currently, we build minimal-notebook again without using cache during push-multi assuming push-multi for base-notebook has already run. It is because we re-tag jupyter/base-notebook:latest I think.

[mathbunnyru]

Related: #1203

[mathbunnyru]

Added this issue to the milestone.

[mathbunnyru]

Another idea is to improve PRs speed - build ARM images only in master or when the commit has some pre-defined string.
This way we will be able to have PRs working at the same speed as before (around 25 minutes).

Also, we might want to use actions/cache.

[manics]

Could you move the multiarch build into a separate GitHub workflow? You'd then get multiple CI statuses on PRs, and could choose to merge after the amd64 job passes instead of waiting for all jobs?

[consideRatio]

@manics I agree that is an important optimization - note that you only need to have separate jobs, not separate workflows (that contains X jobs). I suggest we both separate amd from arm (optimization 2) and separate images from each other (optimization 3).

[consideRatio]

I've ordered 7 RPi computers and look to make them self-hosted arm64 based runners for us in the Jupyter ecosystem where needed.

[mathbunnyru]

I've ordered 7 RPi computers and look to make them self-hosted arm64 based runners for us in the Jupyter ecosystem where needed.

Wow, nice! :)

I also wanted to create some VMs on ARM to use it as self-hosted runners, but if you're already on it, that's great 👍

[mathbunnyru]

@consideRatio did you have any luck with arm runners?

[consideRatio]

@mathbunnyru I have a k8s cluster running on 7 raspberry pi computers etc, but I've failed to deploy the github runner software on k8s still. I'm left quite clueless on what is going on with that and have failed to debug it.

actions/actions-runner-controller#732

[mathbunnyru]

I see. Unfortunately, I have almost zero experience with k8s and absolutely zero experience with self-hosted runners, so I can't help you right now :(

[mathbunnyru]

@consideRatio I noticed, that the build times in master are really slow.
It seems that for multi-arch images push step rebuilds everything from scratch and we do the same thing twice (building the image).
Could you please take a look?

Latest master branch is still running - build step took only 1h 16m 43s, push is already taking almost an hour and not yet finished.
https://github.com/jupyter/docker-stacks/runs/3431621105