Docker runtime build runs out of disk space

[rzikm]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1767366&view=logs&j=50a69d0a-7972-5459-cdae-135ee6ebe312&t=1d3829d7-1a11-547e-8a52-d7bfaeb52b41

(note that the step above erroneously shows as successfully completed, that is on us)

Build leg reported

runtime-libraries stress-http/Build CLR and Libraries

Pull Request

dotnet/runtime#69182

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : ""
}

Additional information about the issue reported

The step in question runs just:

      $(dockerfilesFolder)/build-docker-sdk.sh -t $(sdkBaseImage) -c $(BUILD_CONFIGURATION)

[MattGal]

Hi @rzikm ! For my understanding, can you please give some details on what you believe broke here?

[rzikm]

Not sure, the last line of the docker build command output is

ApplyLayer exit status 1 stdout:  stderr: write /repo/artifacts/obj/coreclr/Linux.x64.Release/tools/superpmi/mcs/mcs.dbg: no space left on device

This has been happening for some time, but the CI leg was broken due to other reasons as well, so we did not notice when this started happening

[MattGal]

OK, I'll take a look. The "Build.Windows.Amd64.VS2022.Pre doesn't exist in pool NetCore1ESPool-Public" is expected since that's an internal image on public; see https://helix.dot.net/ for which images work where.

[MattGal]

OK so what we have here is:

• build.ubuntu.1804.amd64.open images depended on in the helix world, mounting a 512 GB disk on /datadisks/disk1 and putting docker in in /datadisks/disk1/docker
• Now we're using the same image but no special data disk in a 1ES Hosted machine, so /datadisks/disk1/docker is just the main (tiny) disk
• Runtime builds do large complex in-docker builds in the http stress case, that fills this disk.

In the short term, I've asked @rzikm to try 1es-ubuntu-1804-open image, but I think I can trick docker on the image to use the same ephemeral disk that the 1ES Hosted machines use for storing workspaces; this will buy us a (total of) 100 GB for both the workspace, artifacts, and docker containers; a pretty big upgrade.

[MattGal]

I prepped https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/22889 which should move the docker folder from a disk with ~18 GB free to one with ~90 GB free (albeit shared with the build workspace)

[MattGal]

PR is merged; once that's complete we can try @rzikm 's build on the staging environment as an experiment.

[rzikm]

Using 1es-ubuntu-1804-open image seems to fix the issue for us.

[MattGal]

Using 1es-ubuntu-1804-open image seems to fix the issue for us.

Great! I think we will soon also have more docker capacity in the other image.

[MattGal]

Test run to make sure docker still works as expected: https://dev.azure.com/dnceng/public/_build/results?buildId=1770791&view=results (Edit: it does)

[MattGal]

This is rolled out, closing