OSX-based arm64 jobs sending workloads to `osx.1200.arm64.open` are timing out.

[matouskozak]

In dotnet/runtime, OSX-based arm64 jobs sending workloads to osx.1200.arm64.open are timing out.

Started on ±Tuesday (2024/12/31) between 10:00-22:00 as reported in https://dev.azure.com/dnceng-public/public/_build/results?buildId=904732&view=results.

Affecting osx-arm64 NativeAOT and MacCatalyst jobs

Between the last non-failing and first-failing builds, there are no dotnet/runtime commits, hinting at possible infrastructure issue.

[garath]

Agree, telemetry shows machines in that queue stopped taking work right around midnight UTC. We'll investigate.

[garath]

The queue has dropped from 30 machines to 10, but those 10 are responding properly and taking work.

[matouskozak]

The queue has dropped from 30 machines to 10, but those 10 are responding properly and taking work.

@garath thank you for the update, is there a plan to restore the original number of devices or should we align our workloads to fit the reduced queue?

[MichalStrehovsky]

The logs have following message: "Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options."

Is there any guideline on which queue we should move to? I tried looking for a deprecation announcement in my emails but I wasn't able to find mentions of osx.1200.arm64.open.

Cc @dotnet/runtime-infrastructure

[matouskozak]

The logs have following message: "Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options."

Is there any guideline on which queue we should move to? I tried looking for a deprecation announcement in my emails but I wasn't able to find mentions of osx.1200.arm64.open.

Cc @dotnet/runtime-infrastructure

There is a thread in First Responders about this, tagged you.

[garath]

I've tracked down the missing machines as having been reassigned to other queues. Though the timing didn't happen as we expected, this queue has been scheduled for downsizing for a while now. The queue will continue have 10 machines for now (though, as noted, macOS 12 is deprecated, so all machines will eventually be reallocated).

Looking at the telemetry I see that the queue times have been within SLA expectations even with the lowered count. The infrastructure problem that spawned this issue happened because the machines all removed for a few hours (then 10 were returned).

So, we in dnceng will have a look at our processes for any improvements to prevent this sort of "Lots then None then Some" adjustment from happening again. There are no plans to add machines back to this queue. I recommend anyone using this queue look at removing their use (or, get in touch with us if this version is required for your use-case).