Apply arg_replacements scrubbing to params file cache key hashing#30143
Open
oliviernotteghem wants to merge 1 commit into
Open
Apply arg_replacements scrubbing to params file cache key hashing#30143oliviernotteghem wants to merge 1 commit into
oliviernotteghem wants to merge 1 commit into
Conversation
Behavior is conditional to --experimental_remote_scrubbing_param_files_arg_replacements_enabled param
Summary:
Intent:
- The xplat scrubber's `arg_replacements` rules normalize command-line arguments before
they enter the remote cache key, allowing machines with different paths/workspace
roots to share cache hits.
- Params files (virtual action inputs written as `@params` files) were previously hashed
using their raw byte content, bypassing the `arg_replacements` logic entirely.
- This change extends the same normalization to params file content so that the hash
used in the Merkle tree reflects the scrubbed arguments, not the raw ones.
Changes:
- `CommandLines.ParamFileActionInput`: added `getType()` and `getCharset()` getters
so callers can reconstruct a params file with different arguments.
- `DirectoryTreeBuilder.buildFromActionInputs()`: when a `VirtualActionInput` is a
`ParamFileActionInput` and a `SpawnScrubber` is active, each argument is passed
through `spawnScrubber.transformArgument()` before the digest is computed. The
resulting Merkle tree node (both hash and byte content) reflects the scrubbed args,
consistent with how regular command-line args are already handled in `buildCommand()`.
Tags: #has_java
Differential Revision: https://code.uberinternal.com/D25337121
github-actions
Bot
added
team-Performance
fmeum
reviewed
Jul 6, 2026
| public abstract boolean getExperimentalRemoteCacheChunking(); | ||
|
|
||
| @Option( | ||
| name = "experimental_remote_scrubbing_param_files_arg_replacements_enabled", |
Collaborator
There was a problem hiding this comment.
I wouldn't recommend making this a flag, the old behavior is a straight up bug and experimental features can be changed at will anyway.
| scrubbedArgs, | ||
| paramFile.getType()); | ||
| } | ||
| var digest = digestUtil.compute(virtualActionInput); |
Collaborator
There was a problem hiding this comment.
You could avoid eager materialization of all args by adding a writeToTransformed method to ParamFileActionInput that accepts a transformation and then call that function in a lambda DeterminsticWriter in the call to digestUtil.compute below.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Behavior is conditional to --experimental_remote_scrubbing_param_files_arg_replacements_enabled param
Summary:
Intent:
- The xplat scrubber's
arg_replacementsrules normalize command-line arguments before they enter the remote cache key, allowing machines with different paths/workspace roots to share cache hits.- Params files (virtual action inputs written as
@paramsfiles) were previously hashed using their raw byte content, bypassing thearg_replacementslogic entirely.- This change extends the same normalization to params file content so that the hash used in the Merkle tree reflects the scrubbed arguments, not the raw ones.
Changes:
-
CommandLines.ParamFileActionInput: addedgetType()andgetCharset()getters so callers can reconstruct a params file with different arguments.-
DirectoryTreeBuilder.buildFromActionInputs(): when aVirtualActionInputis aParamFileActionInputand aSpawnScrubberis active, each argument is passed throughspawnScrubber.transformArgument()before the digest is computed. The resulting Merkle tree node (both hash and byte content) reflects the scrubbed args, consistent with how regular command-line args are already handled inbuildCommand().Tags: #has_java
Differential Revision: https://code.uberinternal.com/D25337121
Description
Motivation
Build API Changes
No
Checklist
Release Notes
RELNOTES: None