Significantly less available disk space in macos-14-arm64

[ehuss]

Description

The 20240827.4 release of the macos-14-arm64 image seems to have significantly less available disk space than the previous release (20240818.4). I wanted to check if this was intended or expected. This is unfortunately causing our builds to fail.

20240827.4 initial space:

Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1   295Gi   9.6Gi    18Gi    35%    404k  190M    0%   /
devfs            195Ki   195Ki     0Bi   100%     674     0  100%   /dev
/dev/disk3s6     295Gi    20Ki    18Gi     1%       0  190M    0%   /System/Volumes/VM
/dev/disk3s2     295Gi   4.9Gi    18Gi    22%     153  190M    0%   /System/Volumes/Preboot
/dev/disk3s4     295Gi   288Ki    18Gi     1%      23  190M    0%   /System/Volumes/Update
/dev/disk1s2     500Mi    20Ki   495Mi     1%       0  5.1M    0%   /System/Volumes/xarts
/dev/disk1s1     500Mi    96Ki   495Mi     1%      20  5.1M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3     500Mi   196Ki   495Mi     1%      17  5.1M    0%   /System/Volumes/Hardware
/dev/disk3s5     295Gi   261Gi    18Gi    94%    2.2M  190M    1%   /System/Volumes/Data
map auto_home      0Bi     0Bi     0Bi   100%       0     0     -   /System/Volumes/Data/home

compared to 20240818.4 initial space:

Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1   295Gi   9.6Gi    57Gi    15%    404k  593M    0%   /
devfs            195Ki   195Ki     0Bi   100%     674     0  100%   /dev
/dev/disk3s6     295Gi    20Ki    57Gi     1%       0  593M    0%   /System/Volumes/VM
/dev/disk3s2     295Gi   4.9Gi    57Gi     9%     153  593M    0%   /System/Volumes/Preboot
/dev/disk3s4     295Gi   272Ki    57Gi     1%      14  593M    0%   /System/Volumes/Update
/dev/disk1s2     500Mi    20Ki   495Mi     1%       0  5.1M    0%   /System/Volumes/xarts
/dev/disk1s1     500Mi    96Ki   495Mi     1%      20  5.1M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3     500Mi   196Ki   495Mi     1%      17  5.1M    0%   /System/Volumes/Hardware
/dev/disk3s5     295Gi   222Gi    57Gi    80%    2.1M  593M    0%   /System/Volumes/Data
map auto_home      0Bi     0Bi     0Bi   100%       0     0     -   /System/Volumes/Data/home

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240827.4

https://github.com/rust-lang-ci/rust/actions/runs/10612800738/job/29415273318

Is it regression?

Yes, 20240818.4 works. https://github.com/rust-lang-ci/rust/actions/runs/10622377375/job/29446525166

Expected behavior

Would like to have more than 18GiB free when starting.

Actual behavior

Only 18GiB free when starting.

Repro steps

1. Run a job on macos-14-arm64 that uses the new image.

[susmitamane]

Hi @ehuss

We are looking into it. Will keep you posted.

[erik-bershel]

Hey @ehuss!

Please, see the description of GitHub hosted runners bellow:

• https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

Thank you for bringing this fact to our attention. Most likely, it is related to the fact that a new Xcode with all SDKs and simulators was added to the assembly, but this still fits into the stated service agreement and is not a regression. Please, see the description of GitHub hosted runners bellow:

• https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

Just in case, we will of course check what exactly led to such a sharp decrease in space and will try to do something about it, but unfortunately we cannot make any promises. MacOS images are the most voluminous and it is quite difficult to remove something from there without negatively affecting the experience of other users. If your builds require additional space, CPU or RAM, then I recommend paying attention to Larger Runners:

• https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners

[nth10sd]

Is there any equivalent cleanup command as per #10386 but for macOS runners instead?

[workingjubilee]

I appreciate you looking into this, but regarding the recommendation to use larger runners for macOS, I am curious: which one is the one with more than 14GiB storage (as promised by service contract), exactly?

Runner Size	Architecture	Processor (CPU)	Memory (RAM)	Storage (SSD)	Workflow label
Large	Intel	12	30 GB	14 GB	macos-latest-large, macos-12-large, macos-13-large [latest], macos-14-large[Beta]
XLarge	arm64 (M1)	6 (+ 8 GPU hardware acceleration)	14 GB	14 GB	macos-latest-xlarge, macos-13-xlarge [latest], macos-14-xlarge[Beta]

[erik-bershel]

Hey @nth10sd!

We'll definitely propose something like that to you later. It's a good temporary solution I think.

[erik-bershel]

Hey @workingjubilee!

Nice input. I checked up with the corresponding team and it seams that you are right - there is no option to extend disk space for macOS as it might be done for the Ubuntu/Windows images.

Given the above and all the information in general, the best solution would be to try to increase the free space on our side - we will try to work on this. As an interim solution, we will develop a couple of small scripts that can be implemented in your CI in order to free up space on the runner by removing components that are not needed at the moment.

cc: @ehuss

[the8472]

APFS seems to support reflinks/file clones. Has deduplication been applied to the runner images? If not then that might save some space.

[igorGevaerd]

We are suffering from the same issue too