Skip to content

CI: use alternative disks if available #148146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

CI: use alternative disks if available #148146

wants to merge 3 commits into from
+64 −0

Conversation

@the8472
Copy link
Member

@the8472 the8472 commented Oct 26, 2025 •
edited
Loading

cleaning up disk space takes a lot of time, sometimes 30 minutes

@rustbot rustbot added A-CI Area: Our Github Actions CI A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Oct 26, 2025
@the8472 the8472 force-pushed the ci-use-extra-disk branch 2 times, most recently from b040885 to 4848df3 Compare October 26, 2025 17:02
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@the8472 the8472 mentioned this pull request Oct 26, 2025
Closed
@the8472 the8472 marked this pull request as ready for review October 26, 2025 19:04
@rustbot rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 26, 2025
@rustbot rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Oct 26, 2025
@rustbot
Copy link
Collaborator

rustbot commented Oct 26, 2025

r? @marcoieni

rustbot has assigned @marcoieni.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@the8472
Copy link
Member Author

the8472 commented Oct 26, 2025

OLD https://github.com/rust-lang/rust/actions/runs/18821126435
NEW https://github.com/rust-lang/rust/actions/runs/18821542169

image

marcoieni
marcoieni reviewed Oct 27, 2025
View reviewed changes
marcoieni
marcoieni reviewed Oct 27, 2025
View reviewed changes
marcoieni
marcoieni reviewed Oct 27, 2025
View reviewed changes
marcoieni
marcoieni reviewed Oct 27, 2025
View reviewed changes
marcoieni
marcoieni reviewed Oct 27, 2025
View reviewed changes
src/ci/scripts/free-disk-space-linux.sh Outdated
Comment on lines 265 to 272
local blkdev=$(df -k "$mountpoint" --output=source | tail -n 1)
echo "Sufficient space available on $blkdev mounted at $mountpoint"
sudo swapoff -a || true
mkdir ./obj
# remount with O_EATMYDATA while we're at it
sudo umount /mnt
sudo mount $blkdev ./obj -o $mntopts || sudo dmesg | tail -n 20
sudo chown -R "$USER":"$USER" ./obj
Copy link
Member

@marcoieni marcoieni Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add more comments explaining what you are doing here? 🙏

@marcoieni
Copy link
Member

marcoieni commented Oct 27, 2025

I have no objections if this works as it seems faster than freeing up disk space 🙌
Let's make sure to try this on many jobs to prevent the auto build to fail.
I would definitely mark this as rollup never.

the8472
the8472 commented Oct 27, 2025
View reviewed changes
src/ci/scripts/free-disk-space-linux.sh Outdated
# remount under the obj dir which is used by docker scripts to write most
# of our build output. And apply optimized mount options while we're at it.
mkdir ./obj
sudo mount $blkdev ./obj -o $mntopts || (sudo dmesg | tail -n 20 ; exit 1)
Copy link
Member Author

@the8472 the8472 Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added an exit 1 here

Copy link
Member

@marcoieni marcoieni Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry if I took so long reviewing this.

So if the mount fails, CI fails? Shouldn't we fallback to disk space cleanup?

Copy link
Member Author

@the8472 the8472 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hrrm maybe. Something going wrong here is kind of unexpected. It means the mountpoint does exist but the mount options are wrong, e.g. because it's not ext4.
I suppose microsoft could change their default filesystem choice for some reason...

I guess I'll just print a warning and then fallback.

Copy link
Member Author

@the8472 the8472 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed it to warning + fallback

@the8472 the8472 requested a review from marcoieni October 27, 2025 23:00
@marcoieni
Copy link
Member

marcoieni commented Nov 7, 2025

I was also thinking: it's possible to run this github action step in the background, right?
We would wait for it before the docker build start.

Just in case in the future we want to cleanup the disk space in the background 👍

@the8472
Copy link
Member Author

the8472 commented Nov 7, 2025

I was also thinking: it's possible to run this github action step in the background, right?
We would wait for it before the docker build start.

Kinda, but deleting currently takes many more minutes than it takes us to reach the docker step, so it doesn't save that much.
I tried running both directory enumeration and deleting in parallel fashion and it looked like it maybe it cut down times, but the variance from run to run is so high that I'm not sure.

We could even run it in parallel with the docker step too since docker doesn't need the space immediately, but that would risk failures when deleting is really slow, like the 30 minutes I linked at the top.

So switching to a different drive is much faster, as long as it is available.

the8472 and others added 3 commits November 13, 2025 22:40
@the8472
CI: use alternative disks if available
f3202e1 
cleaning up disk space takes a lot of time
@the8472 @marcoieni
[review] add more comments
7d3733d 
Co-authored-by: Marco Ieni <11428655+marcoieni@users.noreply.github.com>
@the8472
print a warning and fallback instead of failing ci
32902a9 
we assume the disks are ext4, but GH doesn't guarantee that.
@rustbot
Copy link
Collaborator

rustbot commented Nov 13, 2025

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcoieni marcoieni Awaiting requested review from marcoieni

Assignees

@marcoieni marcoieni

Labels

A-CI Area: Our Github Actions CI A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@the8472 @rust-log-analyzer @rustbot @marcoieni