Linux arm64 hosted runners now available for free in public repositories (Public Preview) 🎉

[samus-aran]

Now in public preview, Linux arm64 hosted runners are available for free in public repositories. Following the release of arm64 larger hosted runners in June, this offering now extends to the open source-community. ❤️

Powered by the Cobalt 100-based processors, these 4 vCPU runners can deliver up to a 40% performance boost compared to Microsoft Azure's previous generation of Arm-based VMs, providing a power-efficient compute layer for your workloads. Arm-native developers can now build, test and deploy entirely within the arm64 architecture without the need for virtualization on your Actions runs.

How to use the runners

To leverage the arm64 hosted runners, add the following labels in your public repository workflow runs:

• ubuntu-24.04-arm
• ubuntu-22.04-arm

Important

Please note that these labels will not work in private repositories, and the workflow will fail if added.

All runs in public repositories will adhere to our standard runners usage limits, with maximum concurrencies based on your plan type. While the arm64 runners are in public preview, you may experience longer queue times during peak usage hours.

Images

In partnership with Arm, GitHub provides the Ubuntu VM images for these runners, helping customers with a seamless start to building on Arm. To view the list of installed software, give feedback, or report issues with the image, visit the partner-runner-images repository.

Get started today!

To get started, simply add one of the new labels to the runs-on syntax in your public Actions workflow file. For more information on arm64 runners and how to use them, see our documentation.

Share below what open source projects you maintain and how using GitHub Linux arm64 hosted runners will benefit you! Let us know how much speed performance they improve 🏎️ 🏎️

Read GitHub changelog

[bjorn3]

I tried using this in https://github.com/rust-lang/rustc_codegen_cranelift/actions/runs/12814835730/job/35732202343, but it seems that the cargo binary is not marked as executable.

[CodeDoctorDE]

Hello, i have the same issue that the directory permissions are wrong, see https://github.com/orgs/community/discussions/148648#discussioncomment-11858098

[bjorn3]

This looks to be resolved now.

[CodeDoctorDE]

I tested it a bit, but it looks like it uses the wrong permissions:

warning: unable to access '/home/runneradmin/.config/git/ignore': Permission denied
warning: unable to access '/home/runneradmin/.config/git/attributes': Permission denied
warning: unable to access '/home/runneradmin/.config/git/config': Permission denied
warning: unable to access '/home/runneradmin/.config/git/attributes': Permission denied
warning: unable to access '/home/runneradmin/.config/git/ignore': Permission denied
warning: unable to access '/home/runneradmin/.config/git/attributes': Permission denied
warning: unable to access '/home/runneradmin/.config/git/ignore': Permission denied

This is the error why it fails:

Error: Flutter failed to create a directory at "/home/runneradmin/.config".

this is my github action: https://github.com/LinwoodDev/Butterfly/blob/7cb574ed18fea6f8aa825069dcd8955cae99d1e7/.github/workflows/build.yml

https://github.com/LinwoodDev/Butterfly/actions/runs/12815058129/job/35732903060?pr=787

[nazar-pc]

Seems to be fixed, my latest build proceeded past that error now

[balupton]

yep fixed

[raffadh28]

Are there any permissions or user privilege changes required in the GitHub Actions workflow to fix this issue?

[maxim-lobanov]

Hello, rollout of the fix is finished. Let us know if you still see the same issue.

[kjxbyz]

See https://github.com/orgs/community/discussions/149529

[cdce8p]

Is there a plan to add a ubuntu-arm-latest tag similar to ubuntu-latest?

[lkfortuna]

there is no plan to add that at this time.

[xlionjuan]

I think people who using latest knows the risk, but some jobs are too simple that making latest not that dangerous, the more ARM runners are adopted, the more the footprint is reduced, isn't it?

[darthmaim]

The new runners don't report that they are ARM and have incorrect links in the "Set up job" step:

The "Runner Image" should report ubuntu-24.04-arm and the included software should link to https://github.com/actions/partner-runner-images/blob/main/images/arm-ubuntu-24-image.md

[lkfortuna]

Hey @darthmaim! We are aware of this issue and are working on a fix for it. Thanks for reporting!

[FoxxMD]

I'm puzzled about my perceived lack of performance gains using the new runners...

For my project building a multi-platform docker image using GHA with QEMU/buildx/build-push-action on a normal runner has always taken a long time (11-13min) -- and the majority of that time was spent on the arm emulation part of the build. The x86 part takes ~1:30 while arm finishes in 9-10 minutes. I've always expected this would be the case due to the emulation.

But now I've tried the new runner using a GHA where a matrix builds my image on separate runners, arm/x86 native without QEMU, and the difference in build time is only about 8% faster. I'm scratching my head about why this isn't improved more. For an absolute comparison, the project builds the same docker image (for arm platform only) on my Raspberry Pi 5 in 1:40 and 1:50 on my RPI4b (compared to 8:37 on ubuntu-22.04-arm).

Is this expected? Are the new runners this anemic in terms of performance?

EDIT: AS @darthmaim mentions below the solution is to use runs-on: {{ matrix.os }} as well as configuring the matrix to use the correct image/arch/os. Build times are now down to 2min. The working workflow here

[darthmaim]

You have to change your runs-on: ubuntu-latest to runs-on: {{ matrix.os }}.

I have a similar workflow that went from ~14 minutes using QEMU to ~2 minutes using the ARM runner.

[nightlark]

I'm seeing a similar (or better?) performance improvement. I have a Docker image build for arm64 that was taking at 3.5 hours with QEMU, and after switching to the native arm64 runners it finished in 16 minutes.

[raffadh28]

Are there any ways to further improve the build time with the new runners?

[nightlark]

Are there any ways to further improve the build time with the new runners?

The build times I’ve seen so far are the same or slightly faster than the same build on the amd64 Linux runners. What are you looking for? Could always pay for a larger runner instance with more cores if your build/tests can take advantage of the increased parallelism.

[FoxxMD]

Thank you @darthmaim that did the trick. Down to 2 minutes for execution on the arm build now 👍

[kjxbyz]

See https://github.com/orgs/community/discussions/149347

[lkfortuna]

hey @kjxbyz - we are aware of the issue and are in process of rolling out a fix

[kjxbyz]

If the issue is fixed, please reply in this discussion. Thank you!

[maxim-lobanov]

Hello, rollout of the fix is finished. Let us know if you still see the same issue.

[Athou]

The ubuntu-24.04-arm runner seems to miss the libgbm1 package:

╔══════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ Host system is missing dependencies to run browsers.                                                 ║
║ Please install them with the following command:                                                      ║
║                                                                                                      ║
║     sudo mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install-deps" ║
║                                                                                                      ║
║ Alternatively, use apt:                                                                              ║
║     sudo apt-get install libgbm1                                                                     ║
║                                                                                                      ║
║ <3 Playwright Team                                                                                   ║
╚══════════════════════════════════════════════════════════════════════════════════════════════════════╝

See https://github.com/Athou/commafeed/actions/runs/12823180867/job/35757289975

The package is installed on ubuntu-latest. I fixed this with this but it would be nice if the packages installed on both ubuntu runners were the same 😊

  - name: Install Playwright dependencies
    run: sudo apt-get install -y libgbm1
    if: matrix.os == 'ubuntu-24.04-arm'

[lkfortuna]

Hi there- for image related feedback, please open an issue in this repo! https://github.com/actions/partner-runner-images

[jacob-carlborg]

Would have been nice to have support for hardware accelerated nested virtualization (KVM). Seems it's not available:

$ sudo modprobe kvm
$ sudo /usr/sbin/kvm-ok
INFO: /dev/kvm does not exist
HINT:   sudo modprobe kvm
modprobe: FATAL: Module msr not found in directory /lib/modules/6.8.0-1017-azure
$ sudo dmesg | grep -i kvm
[    0.585208] kvm [1]: HYP mode not available

Does anyone know if there's a limitation in hardware, the virtualization layer or if it's just not configured to be enable?

[lkfortuna]

hey there- nested virt is not supported for this sku https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dpdsv6-series?tabs=sizebasic#feature-support

[jacob-carlborg]

That's disappointing 😞, but thanks for letting us know.

[lkfortuna]

as soon as azure supports it, we can support it, but just blocked until then.

[orazioedoardo]

No ubuntu-20.04-arm runner? Technically it's still supported.

EDIT: Ok, saw this as well https://github.blog/changelog/2025-01-15-github-actions-ubuntu-20-runner-image-brownout-dates-and-other-breaking-changes/

[dalcinl]

From time to time, actions/checkout@v4 is failing but I see no output, for example:
https://github.com/mpi4py/mpi4py-publish/actions/runs/12827690739/job/35770199992#step:2:1

I'm also seeing some docker instabilities when running jobs that have large matrices. Typical error messages are:

https://github.com/mpi4py/mpi4py-publish/actions/runs/12827690739/job/35770200582#step:10:398

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-8fd5c4828e7358dfa718c5dddf5f48350fc6a3e059fba62947fb8c540ddbf3fd.scope" (properties [{Name:Description Value:"libcontainer container 8fd5c4828e7358dfa718c5dddf5f48350fc6a3e059fba62947fb8c540ddbf3fd"} {Name:Slice Value:"system.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [4710]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms): unknown```

https://github.com/mpi4py/mpi4py-publish/actions/runs/12827690739/job/35770199123#step:10:287

+ docker version -f '{{json .}}'
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[dalcinl]

Here you have an example of actions/checkout@v4 failing 4 times in a matrix of 48 rows:
https://github.com/mpi4py/mpi4py-publish/actions/runs/12842120486/attempts/1

[sterliakov]

+1, I also reported this to actions/checkout (actions/checkout#2064, with another workflow and relevant logs), but probably there is some issue to solve on the runner level.

[brandond]

For the Is the docker daemon running? error, we've had to add a while ! docker version; do sleep 1; done to the start of our arm jobs. It seems like these nodes are a good bit slower and docker isn't up yet when the step runs some portion of the time.

[lkfortuna]

Hey there- we are still investigating but we believe if you use the ubuntu-22.04-arm runners this will fix the issue, do you mind testing on that runner?

[pllim]

Hello. I ran into permission problem using this new runner with tox. This same workflow was using paid linux-arm64 before and did not have such problem. Please see:

• TST: Use free Linux ARM64 runner astropy/astropy#17645

[lkfortuna]

the permission issue has been fixed

[nazar-pc]

Should have been called -arm64 or -aarch64

[eregon]

Agreed, it's odd to have macos-14-arm64 but ubuntu-24.04-arm.
Also Node.js os.arch() is arm64, as is core.platform.arch.

[nazar-pc]

uname -p is aarch64, so are many Linux packages and other things, but arm is the worst possible option, it is inconsistent with basically everything

[eregon]

Yeah, arm sounds like 32-bit ARM which is not the case.
Even the first post above mentions "arm64" 7 times.

FWIW I tried ubuntu-22.04-arm/ubuntu-24.04-arm64 but those are not recognized.
Would be nice to at least make them aliases.

[webknjaz]

$ python3.13 -c 'import platform;print(platform.machine())'
arm64

[mhemani-reach]

Are there plans for a ubuntu-latest-arm label?
Are there plans to allow free usage in private repos (especially for Enterprise Cloud)?

[CurryYangxx]

Since we can't use it in the private repo now, is there an alternative solution now?

[LesnyRumcajs]

@CurryYangxx If you're willing to pay, there are buildjet or warpbuild runners (and probably lots of others). I've used buildjet in the past for ARM builds and it was okay, but I'm glad I could switch to fully GH infra.

[lkfortuna]

We do offer them in private on our larger runners, they just do not consume entitlement minutes.

[lkfortuna]

https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners

[brentisaacs]

Hello, @lkfortuna, do you know if the release of Arm runners to Private repos is still slated for 2025H2? I heard 2025Q2 in another location, so I wanted to get clarity on how long we should expect to run our own solution in the meantime.

[dennisameling]

Ran into this error while testing the new ubuntu-22.04-arm runners:

JavaScript Actions in Alpine containers are only supported on x64 Linux runners. Detected Linux Arm64

Looks like there's an open issue for this since 2020. Any chance this could be prioritized? Thanks!

[userdocs]

Until you can run commands on the host before the container is set up the system is crippled anyway.

You have always been able to call a docker image on demand from the runner and not use container:

https://github.com/userdocs/test/actions/runs/12828579208/workflow

As you can see build-3 does not fail, which is why I don't have this issue.

[eregon]

I've never seen this error before, only on these new linux-arm64 images:
https://github.com/ruby/ruby-builder/actions/runs/12844097279/job/35816748560

Run gh release upload "toolcache" "ruby-3.2.4-ubuntu-22.04-arm64.tar.gz"
failed to create root command: failed to read configuration: open /home/runneradmin/.config/gh/config.yml: permission denied

Restarting the failed jobs is a workaround but obviously not ideal.

[lkfortuna]

This was related to the permissions issue we had last week - we rolled out a fix so that is why you saw this succeed on retry. It should not be an issue going forward.

[mkeeter]

I've seen significant unreliability with the ubuntu-24.04-arm runners while building and checking Rust code. For example, here's Clippy failing with an internal compiler error (?!). I have also seen rustc crash in similar ways.

Rerunning the jobs succeeds, and aarch64-unknown-linux-gnu is a Tier 1 target for Rust, so I suspect the runner is at fault.

I'm not sure how the runner could induce this particular error, though, besides memory corruption:

thread 'rustc' panicked at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/compiler/rustc_serialize/src/opaque.rs:269:45:
range start index 87597202347078788 out of range for slice of length 40857713

[edmorley]

We've been seeing this too, as have a number of others :-(

See also:

• ICE: rmeta/decoder.rs:967:9: missing def_span for DefId(117:0 ~ test[50b7]) rust-lang/rust#135939
• Compiler crashes/ICEs on new aarch64 GHA runners/Azure Cobalt 100 (Neoverse N2) CPUs rust-lang/rust#135867
• test-fast on ubuntu-24.04-arm intermittent SIGSEGV or SIGBUS in rustc GitoxideLabs/gitoxide#1790
• ubuntu-24.04-arm: Python Program terminated with signal SIGILL, Illegal instruction actions/partner-runner-images#36
• https://github.com/orgs/community/discussions/149715

What's strange is that we didn't see these issues when using ARM on GHA using the "larger runners" feature. I had thought the new public runners would be using the same image types (/machine types), but it seems there are differences? For more on this, see:
rust-lang/rust#135939 (comment)

[edmorley]

It looks like this might be related to the ubuntu-24.04-arm runners using the Cobalt 100 processor (Neoverse N2), whereas the larger runners are apparently using the Neoverse N1:
actions/partner-runner-images#36 (comment)

[edmorley]

See also https://support.github.com/ticket/personal/0/3198300

[lkfortuna]

hi, were taking a look at this one

[lkfortuna]

Hey there- we are still investigating but we believe if you use the ubuntu-22.04-arm runners this will fix the issue, do you mind testing on that runner?

[Firefishy]

Does anyone have a elegant & clean method for creating multi-platform docker image? (ARM64 + AMD64).

I'm quietly hoping that Docker come-up with an addition to their actions for the task. eg: New addition to docker/metadata-action, docker/build-push-action etc to help with multi-platform job builds.

I've seen https://github.com/gartnera/actions-arm64-native-example and while it does work, it doesn't feel elegant to me.

[xlionjuan]

You could check mine, though looks not elegant, but it is needed, because you've separated different runners(jobs), you need a way to link the tags.

https://github.com/xlionjuan/fedora-createrepo-image/blob/main/.github%2Fworkflows%2Fbuild_all.yml

[flaviomartins]

This is mine https://github.com/bdist/notebook/blob/main/.github/workflows/docker-publish.yml

PS: let me know if there are any bugs 👍

[samus-aran]

Hi everyone! We’ve resolved the recent permissions issue, and everything should now be working as expected. If you’re still experiencing any problems with permissions, please don’t hesitate to reach out – we’re here to help! 👍

[tisba]

Might be because of high demand / public preview, but for mini_racer I could not get a job scheduled to a worker ("Waiting for a runner to pick up this job") https://github.com/rubyjs/mini_racer/actions/runs/12908769152/job/35994991735?pr=341. Waiting since 2 hours 😢

[maxim-lobanov]

Hi 👋 ,
I think you have a typo in runs-on. it should be ubuntu-24.04-arm, not ubuntu-24.04-arm64

[tisba]

arg damn 🤦 thx!

[tisba]

although, arm64 would be much more consistent, or -aarch64. In any case, not throwing an error is kind of bad too

[jauderho]

I currently build multi-arch Docker images on x86 runners like so: https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/age.yml

Given that arm64 is available, I'd rather build the arm specific ones on arm rather than through qemu. Looking for ideas how to accomplish this without having to massively rework the workflow file.

[nightlark]

You could use a build matrix and define the platforms to use there. The changes would be something like...

Replace https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/age.yml#L23 with:

    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        include:
          - os: ubuntu-24.04
            platforms: linux/amd64,linux/ppc64le,linux/s390x
          - os: ubuntu-24.04-arm
            platforms: linux/arm64,linux/arm/v7,linux/arm/v6

Replace https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/age.yml#L93C22-L93C97 with:

          platforms: ${{ matrix.platforms }}

[jauderho]

@nightlark

I was hoping this would work but as I suspected, the :latest tags and the version tags are getting overridden as one side completes instead of getting appended. Not quite sure how to fix this.

See

• https://github.com/jauderho/sandbox/blob/main/.github/workflows/age.yml
• https://github.com/jauderho/sandbox/pkgs/container/age-sandbox/352185598?tag=v1.2.1

Looking at https://docs.docker.com/build/ci/github-actions/multi-platform/ under "Distribute build across multiple runners", there looks to be a workaround but it sure looks inelegant.

[Ferroin]

Looking at https://docs.docker.com/build/ci/github-actions/multi-platform/ under "Distribute build across multiple runners", there looks to be a workaround but it sure looks inelegant.

While it looks inelegant, I can attest that it does work (provided you don’t hit the stability issues that seem to be plaguing the Ubuntu 24.04 ARM images), is easily scalable, allows for easy mixing of native/QEMU builds (this matters if you want to build for platforms other than just x86 and ARM, such as POWER or RISC-V), and also allows decoupling publishing to multiple registries.

[jauderho]

BTW does anyone have any data showing the performance of the new ARM systems compared to the x86 ones?

[dennisameling]

BTW does anyone have any data showing the performance of the new ARM systems compared to the x86 ones?

When compiling NodeJS for Linux arm64, it seems to be slightly faster than its x64 equivalent. Here's a workflow run that had a 0% cache hit for both, allowing for easy comparision.

• x64 build time: 2h 0m 25s
• arm64 build time: 1h 52m 5s

While the arm64 build was slightly faster, it spent more time in the Test step than the x64 version, so the total workflow run was slightly longer (2h 33m 19s) than the x64 one (2h 31m 29s).

[userdocs]

In this current status of all no issues https://www.githubstatus.com

ubuntu-24.04-arm is very fragile in matrix jobs. A minimum of 1 to 3 jobs are failing at basic things like running an action.

This happens every time the matrix is run without fail.

It needs to be resolved for these runners to be reliably used.

[xlionjuan]

I got random error on checkout and QEMU too, it is not really annoying for me, at least it failed at very early state.

[userdocs]

@xlionjuan I am just updating here with the errors I see. Maybe you can do the same. There is clearly some unresolved issue here.

actions/partner-runner-images#37

[lkfortuna]

ah youre right, doesnt look like tickets are showing up in your view, sorry about that. We will take a look here.

[erickguan]

I added a maybe similar problem here. In particular, rustc on the ubuntu-24.04-arm runner when compiling CRuby.

[lkfortuna]

Hey there- we are still investigating but we believe if you use the ubuntu-22.04-arm runners this will fix the issue, do you mind testing on that runner?

[dwoz]

Sometimes docker daemon is running, sometimes it is not running. We're only seeing this on the arm64 runners.

https://github.com/saltstack/salt/actions/runs/13037843628/job/36389023627?pr=67129

[mitesch]

Hi, just want to reiterate that we're actively looking at this. We believe we've isolated it to a particular azure sku/configuration that is experiencing the issue.

If anyone has a very simple workflow that repros. they are always helpful as well.

@jakob-keller yours may be a separate issue. Can you dig in and see if there's any reason for your code/test to be flaky? It also fails when we manually ran it on our known-good sku of arm64, so it may just be something related to building/testing on a different architecture in general.

[xlionjuan]

It may failed even on checkout or Login Docker Hub, and almost can be fixed by rerun.

[consideRatio]

Yepp for https://github.com/jupyter/docker-stacks, we see failures in various places, sometimes related to no docker daemon running, but also often even before when actions/checkout is run (then we see no error message, the step just fails).

[jakob-keller]

@jakob-keller yours may be a separate issue. Can you dig in and see if there's any reason for your code/test to be flaky? It also fails when we manually ran it on our known-good sku of arm64, so it may just be something related to building/testing on a different architecture in general.

By far the most common failure occurs in a single unit test that attempts to spin up a container. It fails in more than 50% of cases. Given enough retries, it'll eventually succeed. The failure rate is 0% for amd64.

I have zero issues running the test suite locally (Apple silicon) when docker is running and see the exact same error whenever my local docker is switched off. That is to be expected, of course.

My best guess is that the docker daemon is somehow unavailable on a large subset of runners. Could that somehow be a shared resource across jobs that suffers from congestion when other users share the public runner?

I will try to produce more details relating to the issue. Stay tuned...

[jakob-keller]

OK, now I believe @mitesch was right after all: The failures are caused when trying to spin up a Lambda container through moto. For some reason, the container image to be launched is picked randomly from a set of options. One of the candidate images does not provide an arm64 variant. That triggers an exception and causes the undeterministic behavior for us.

[LesnyRumcajs]

After switching to ubuntu-24.04-arm we get quite a lot of failures on checkout@v4. It wasn't the case for ubuntu-24.04.

Samples:

https://github.com/ChainSafe/forest/actions/runs/13176064234/job/36775533964?pr=5245

Workflow https://github.com/ChainSafe/forest/blob/6ba1931312430901371694ec14e1ea101dfdc683/.github/workflows/butterflynet.yml

https://github.com/ChainSafe/forest/actions/runs/13176045079/job/36775467670?pr=5247

Workflow https://github.com/ChainSafe/forest/blob/25627734910c5d3bd8587b488caf6477fad07bb4/.github/workflows/scripts-lint.yml#L52-L71

[lkfortuna]

Hey there- we are still investigating but we believe if you use the ubuntu-22.04-arm runners this will fix the issue, do you mind testing on that runner?

[LesnyRumcajs]

I'll let you know if we still have the same issues under that runner. Thanks.

[Rhys-T]

I've switched my repo from ubuntu-24.04-arm to ubuntu-22.04-arm, and so far it seems more stable. Haven't seen it get these errors there yet, though it hasn't been using that runner for very long. I'll reply here again if I see it have problems with that runner.

(I did see one reference to the 22.04 runner also failing - actions/partner-runner-images#37 (comment) - but the same user's comments above seem to suggest that that turned out to be an unrelated issue.)

[jakob-keller]

(I did see one reference to the 22.04 runner also failing - actions/partner-runner-images#37 (comment) - but the same user's comments above seem to suggest that that turned out to be an unrelated issue.)

That is correct! Whatever I wrote with regards to ubuntu-22.04-arm was invalid.

[LesnyRumcajs]

@lkfortuna I can confirm that downgrading our workflows to ubuntu-22.04-arm made the checkout errors go away.

[consideRatio]

Issues with arm ubuntu 2204 and 2404 runners have gone away for jupyter/docker-stacks, no longer failing on actions/checkout and such! Nice!!

[mathbunnyru]

Update: unfortunately, still happens sometimes on 24.04: https://github.com/jupyter/docker-stacks/actions/runs/13287528395/job/37099753888?pr=2209

[lkfortuna]

Hey Everyone 👋 Thanks for your patience with the checkout errors and random failures. We discovered it was an issue with the underlying hardware and have migrated all the runners to a different arm sku (dpdsv5) and this should solve the errors you were seeing. Anyone who migrated to ubuntu 22 to solve the issue should be able to move back to ubuntu 24 with no problems as well. We reported the issues to the Microsoft Azure team and they are working to debug them, but in the meantime we have moved our runners so that you can use the runners without issue.

[martin-frbg]

Is it possible that the sku change could have led to ubuntu-24.04-arm jobs now executing on Neoverse N1 like the "large" ones instead of Cobalt100 ?
I've been using this setup successfully for CI testing OpenBLAS builds, but since yesterday builds for NeoverseN2 fail with illegal instruction errors during runtime tests, and querying the cpuid returns the one matching N1

[brandond]

It was mentioned in the comment that you are replying to that the runners now use dpdsv5 instance types.

Relying on CI to use any specific model or chipset is probably a bad idea though.

[martin-frbg]

Sure, but it is probably not ideal to have a blog post specifically announce availability of that specific model (NeoverseN2) under a certain job name that then leads to execution on different hardware - it would certainly be nice to know if the change is intentional
(end of the public preview or whatever).
(The VMs were not reliably reporting their cpuid early on, which is why I had resorted to specifying the build target instead of allowing the build script to autodetect it. Now that I have switched back to autodetection, it is definitely seeing NeoverseN1 since yesterday, which would match the Ampere Altra of dpdsv5.)

[lkfortuna]

Yes this was a purposeful change, we migrated the runners back to the v5 sku due to the errors mentioned in the issue comments above. The N2 machines were not working and causing failures so we have migrated back to N1 in order to unblock people from using the arm64 runners.

The change is intentional and like I mentioned in the comment above, we have reported the issues to Microsoft azure team, but until they can resolve them we will not be migrating these runners back.

[martin-frbg]

Thank you very much for this clarification.

[thesuperzapper]

I'm not sure how it would work on the backend, but the ability to run service containers on these new ARM runners, and access them from an X86 job would enable very simple multi-arch docker builds.

This would allow a docker buildx builder running on the ARM node to be accessed from the X86 node using a remote driver, and you can even combine multiple builders using docker buildx create --append ... on the X86 node.

Obviously, you could do something like run a VPN across your various jobs so they can talk to each other, but that does not account for the locality of the actually servers running these jobs, plus, having a job that only runs the builder service causes all sorts of "retry" issues.

[brandond]

Why would you need that? Just build the image for each arch separately and then merge them in a manifest step. Having the runners communicate between each other seems like an unnecessary additional complication.

[consideRatio]

For reference, building and combining arch-specific images and combining them is done in jupyter/docker-stacks repo

[baentsch]

Is this the right place to report issues with arm-runners? Two weeks ago, everything was working well, a specific job completed in 10 mins and now, it simply ceases to progress after different times in different run attempts. Github status is green. Sample job: https://github.com/open-quantum-safe/oqs-demos/actions/runs/13421069421/job/37692339114. Thanks for checking/letting me know where (else) to report this.

[lkfortuna]

this is the correct place, unless its an issue with the image itself in which case the images repo referenced above would be right. in this case though i think the issue is with the docker action itself since they just released a new version and it appears your workflow is using the latest version. Can you report this to the docker team that manages that action?

[baentsch]

Can you report this to the docker team that manages that action?

Thanks for the feedback! This raises 2 questions for me: 1) Who is this docker team/where can I reach them? 2) It does not seem to be a problem with the docker image but rather the way things are set up within GH: Apparently, this action (runner?) has no internet connectivity as the downloading of data from a public server fails just for arm64 (works OK for x64).

[lkfortuna]

hey @baentsch - we are trying to recreate on our end first to see if we can help narrow it down but we are unable to reproduce the error. Can you reproduce this reliably and send us the links to the runs?

[baentsch]

Thanks @lkfortuna for reaching out: https://github.com/open-quantum-safe/oqs-demos/actions/runs/13421069421 is the job that failed 2 times so far (the "sibling" x64 job always passes). Just triggered another re-run to see whether this qualifies for a "reliable reproducer". Let's see...

[baentsch]

Third time, the job passed. So, no reproducibility, sorry. Case closed but thanks for taking a look!

[lkfortuna]

Hey Everyone 👋 Another update here. Azure has fixed the issue with their dpdsv6 hardware. We have tested the fix and the errors we were seeing before are resolved.

We will be rolling out the dpdsv6 hardware to our public arm runners starting tomorrow. This will be a gradual rollout so your runs may not be moved onto the new hardware right away. If you encounter any issues, please continue to report them here. Thank you.

[lakshanthad]

Hey @lkfortuna. Thanks a lot for this! We have already deployed this in our production environments and they are working great so far!
ultralytics/ultralytics#19596

[Bnaya]

Is there ETA when arm is gonna be supported in regular private repos?

[lkfortuna]

https://github.com/orgs/community/discussions/148648#discussioncomment-11920086

[snadampal]

Hi @lkfortuna Are the Neoverse N2 instances enabled for ubuntu-22.04-arm runners as well? I don't see them in one of the repos using the ubuntu 22.04 runners. so, i believe they are not yet enabled. Is there any ETA for this?

[lkfortuna]

we have only rolled out to 50% so not all jobs are running on N2 right now.
We found that not all VMs were patched with the fix so we have paused the rollout and have not fully migrated all the runners to N2 yet.

[snadampal]

thank you! please post here when N2 is enabled for ubuntu-22.04-arm runners as well. We will check our CI then.

[snadampal]

I see N2 instances for ubuntu-22.04-arm runners. thank you!