build: fix disk space exhaustion in release builds#10306
build: fix disk space exhaustion in release builds#10306Roasbeef merged 3 commits intolightningnetwork:masterfrom
Conversation
Roasbeef
added
enhancement
Summary of ChangesHello @Roasbeef, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses critical disk space exhaustion issues that have previously caused release build failures. It introduces a more comprehensive cleanup strategy within the GitHub Actions workflow, including the removal of additional large toolsets and caches, and integrates a step to clear the Go build cache between platform builds. These changes aim to ensure more stable and successful release processes by optimizing disk usage. Highlights
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces several changes to mitigate disk space exhaustion during release builds. The main strategies are to enhance the cleanup-space action with more aggressive cleaning of packages, caches, and Docker images, and to clear the Go build cache between building for different platforms. The changes are logical and should be effective. I've added a couple of suggestions to the cleanup-space action to use more modern commands and remove a redundant step, which will make the script cleaner and more robust.
Roasbeef
requested review from
a team,
bhandras and
bitromortac
and removed request for
a team
In this commit, we significantly expand the cleanup-space GitHub Actions workflow to free up substantially more disk space on GitHub runners. The previous cleanup only removed three large toolsets (dotnet, android, ghc), which should free ~14GB. This enhancement adds removal of several additional large packages and caches, bringing the total freed space to approximately 20-25GB. The specific additions include removing Swift and Julia language runtimes, the hosted toolcache directory, all Docker images, numerous large apt packages (aspnetcore, llvm, php, mongodb, mysql, azure-cli, browsers, and development tools), and various cache directories. We also add disk space reporting before and after cleanup to provide visibility into how much space is actually being freed during workflow runs. This enhancement was motivated by release builds running out of disk space when building for all 15 supported platforms (darwin, freebsd, linux, netbsd, openbsd, windows across multiple architectures). The sequential builds with verbose output were consuming more space than the basic cleanup could provide.
In this commit, we replace the basic inline cleanup command in the release workflow with the comprehensive cleanup-space action that was previously only used in the main CI workflow. The previous release workflow cleanup simply removed the hostedtoolcache directory, which freed only a few gigabytes and proved insufficient for multi-platform release builds. By switching to the cleanup-space action (now enhanced to free 20-25GB), the release workflow will have substantially more disk space available before beginning the build process. This should resolve the disk space exhaustion issues that were occurring during the Windows ARM build phase, which is one of the final platforms in the 15-platform build sequence.
In this commit, we add a call to "go clean -cache" after each platform build in the release script to prevent the Go build cache from accumulating unbounded disk space during the sequential 15-platform build process. When building for multiple platforms in sequence with "go build -v", Go creates intermediate build artifacts and caches compiled packages for each target platform. While this caching improves build performance within a single platform build, it causes the cache to grow substantially when building for many platforms sequentially. With 15 different platform/ architecture combinations, each with their own cached artifacts, this accumulation was contributing to the disk space exhaustion. By clearing the build cache after each platform completes, we prevent this unbounded growth while still allowing each individual platform build to benefit from caching during its own compilation. The module cache is preserved (we only clear the build cache), so dependencies don't need to be re-downloaded between platforms.
Roasbeef
force-pushed
the
free-release-space
branch
from
68da7ce to
dfee570
Compare
| sudo rm -rf /opt/ghc | ||
| # Remove large packages. | ||
| sudo rm -rf /usr/share/swift | ||
| sudo rm -rf /usr/local/julia* |
There was a problem hiding this comment.
Are these things really on the runner?
4 participants
This PR aims to fix an issue that caused the last release build to fail: running out of disk space before the final platform is built.
We do a few things to attempt to reduce disk space: