Clone files into the disk cache where the filesystem supports it - #30776
Open
ladd wants to merge 1 commit into
Open
Clone files into the disk cache where the filesystem supports it#30776ladd wants to merge 1 commit into
ladd wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
ladd
force-pushed
the
disk-cache-cow-upload
branch
2 times, most recently
from
August 18, 2026 20:42
4f477a8 to
0e3a102
Compare
fmeum
reviewed
Aug 19, 2026
| } | ||
|
|
||
| /** Flushes a file's contents to stable storage, where the filesystem supports it. */ | ||
| private static void syncFile(Path path) throws IOException { |
Collaborator
There was a problem hiding this comment.
Please use the same mechanism as in the fallback path (e.g. FileInputStream).
Author
There was a problem hiding this comment.
Something like the revised version? I'm not entirely clear on the importance of the sync semantics here.
Collaborator
There was a problem hiding this comment.
Me neither, which is why I like this better since the semantics agree ;-)
Uploading a file to the disk cache streamed its bytes into a temporary file, so the cache entry became a second physical copy of an output that was already on disk. Copy through FileSystemUtils#copyFile instead: it reaches Files#copy, which lets the filesystem serve the copy as a copy-on-write clone (clonefile on macOS, copy_file_range on Linux with a supporting filesystem), and the entry then shares its blocks with the output it came from. Where no clone is possible -- a different filesystem, or one without support -- Files#copy falls back to a byte copy, so this is never slower than the stream it replaces. The materialization direction already clones, since commit ec90e05 ("Optimize file copies by using NIO methods"). Measured on macOS/APFS with a large iOS app: materializing bazel-out from a warm disk cache costs 200 MiB of real disk for a 2.07 GiB tree, while a locally built app leaves a 13.1 GiB second copy in the cache. This change removes that copy. A clone inherits the source's permissions and mtime, so both are reset: an entry must stay readable for every user of a shared cache, and the mtime is what the garbage collector reads to find the least recently used entries. The cloned temporary file is fsynced before the rename, keeping the durability the streamed write had. Note that a cache entry sharing blocks with an output is no longer freed by deleting either one alone, and that the garbage collector sizes entries by their logical length, so it now over-counts what deletion will reclaim.
ladd
force-pushed
the
disk-cache-cow-upload
branch
from
August 19, 2026 20:09
0e3a102 to
4c41c03
Compare
ladd
marked this pull request as ready for review
August 19, 2026 20:17
github-actions
Bot
added
team-Remote-Exec
fmeum
approved these changes
Aug 19, 2026
Collaborator
|
@bazel-io fork 9.3.0 |
|
✅ Bazel docs preview is ready! Preview URL: https://bazel-pr-30776.mintlify.app/ Updated for |
github-actions
Bot
added
the
community-reviewed
tjgq
approved these changes
Aug 20, 2026
tjgq
added
awaiting-PR-merge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
DiskCacheClient.uploadFilenow copies throughFileSystemUtils#copyFile, which reachesFiles#copyand lets the filesystem serve the copy as a copy-on-write clone:clonefileon macOS,copy_file_rangeon Linux with a supporting filesystem. The cache entry then shares its blocks with the output it came from. The previous code streamed the bytes into the temporary file.Files#copyfalls back to a byte copy, so this is never slower than the stream it replaces.save()helper, shared by the stream path and the new file path.0555and can be older than the upload, an entry has to stay readable for every user of a shared cache, and the garbage collector reads the mtime to find the least recently used entries.Motivation
Materializing outputs from the disk cache already clones, since ec90e05 ("Optimize file copies by using NIO methods"). The upload direction still wrote a second physical copy of a file that was already on disk.
Measured on macOS/APFS: one genrule producing a 1 GiB output, fresh disk cache, real disk consumed via
df.Physical extent mapping (
fcntl(F_LOG2PHYS_EXT)) on the same pair agrees: 0/3 sampled extents shared before, 3/3 after. On a large iOS app build, the disk cache held 13.1 GiB duplicating content that was also present in the output tree.Two consequences worth knowing:
DiskCacheGarbageCollectorsizes entries by their logical length, so it over-counts what deletion reclaims.Build API Changes
No
Checklist
Release Notes
RELNOTES: Files uploaded to the disk cache are now copied with a copy-on-write clone where the filesystem supports it, so a cache entry can share its blocks with the output it was uploaded from.