Windows build of plugins don't start on ci

[gounthar]

Service(s)

ci.jenkins.io

Summary

Following my PR on the antexec plugin, https://ci.jenkins.io/job/Plugins/job/antexec-plugin/view/change-requests/job/PR-110/1/ is taking forever to get a Windows VM.

In progress 1 hr 22 min ago in 1 hr 22 min and counting

@timja has detected the same kind of issues.

Reproduction steps

No response

[dduportal]

Workin on it: requires to block all builds on ci.jenkins.io (put in the queue)

[dduportal]

Ok, there is an ongoing Azure network outage which causes these troubles:

This outage was only notified in our Azure Service Health console today (10 January) around 10:30 UTC, and it looks like it's not being reported properly (look at the map below):

[dduportal]

• status.jenkins.io updated (PR: open Azure + ci.jenkins.io Windows containers outage status#577). Permalink to the incident: https://status.jenkins.io/issues/2025-01-08-ci.jenkins.io-azure-outage/
• Banner added to ci.jenkins.io:

[dduportal]

Since this incident has been ongoing unoticed by Microsoft since the past 2 days, we most porbably don't see any full recovery until a few days. Let's wait and see as the team is not available to perform a full region change for the time being and it's not critically blocked.

[dduportal]

Also did an email to the developers mailing list for awareness: https://groups.google.com/g/jenkinsci-dev/c/VQcRiUYu92o

[lemeurherveCB]

Workaround for stuck builds: set useContainerAgent to false in your pipelines as Azure VM agents are spawned successfully since they're not using the impacted ACI service.

[rantoniuk]

• Banner added to ci.jenkins.io:

Unfortunately that banner is not visible on the actual build pages, the user needs to be on the main Jenkins site to see it...

And it seems it's a good feature request candidate :-)

[lemeurherveCB]

Good point, unfortunately we don't have anything allowing to display a top banner on every ci.jenkins.io pages AFAIK.

[jglick]

(FTR, noticed earlier in jenkinsci/jenkins-test-harness#893)

[jglick]

a full region change

Or just a zonal change? According to the announcement image, other zones in the same region are unaffected.

[jglick]

set useContainerAgent to false in your pipelines

Requires temporarily editing Jenkinsfile and then reverting the edit in each affected plugin. Also there is no need to touch Linux configuration, only Windows, IIUC. Would be nicer to developers to temporarily edit buildPlugin.groovy to either suppress Windows ACI builds, or quietly switch them to VMs.

[lemeurherveCB]

Would be nicer to developers to temporarily edit buildPlugin.groovy to <...> quietly switch them to VMs.

Implemented by @MarkEWaite in jenkins-infra/pipeline-library#898

jenkins-infra/pipeline-library#899 opened as

plugins that use spotless will fail their builds on a Windows VM when they pass on a Windows container. Switch from container to VM makes that issue visible to more people