You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
filegroups are an easy way of grouping files together and using them across several spawns. If a filegroup is used across several spawns (especially if those are executed as persistent workers thousands of times), it increases the size of execution_log_json_file.
We run into this when we use rules_js with:
js_binary(
name = "bin",
data = [
"//:node_modules/react",
"//:node_modules/eslint",
"...",
],
entry_point = "index.js",
)
These node_modules collectively can be hundreds of thousands of files, which multipled with thousands of persistent worker invocations runs easily into millions of lines in execution_log_json_file with the same repeating inputs.
One simple way to address this is to use filegroup if a filegroup appears only once in the execution_log:
filegroup(
name = "node_modules_group",
srcs = [
"//:node_modules/react",
"//:node_modules/eslint",
"...",
]
)
js_binary(
name = "bin",
data = [":node_modules_group"],
entry_point = "index.js",
)
Which category does this issue belong to?
Performance
What underlying problem are you trying to solve with this feature?
Reduce the size of execution_log_json_file in cases where several filegroups are used.
Which operating system are you running Bazel on?
MacOS
What is the output of bazel info release?
release 6.1.0
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
The text was updated successfully, but these errors were encountered:
We are planning to address the shortcomings of the execution log format in Q4 (specifically, by designing a leaner format that avoids storing duplicate information). Therefore I'm going to close this in favor of #18643.
Description of the feature request:
filegroups are an easy way of grouping files together and using them across several spawns. If a filegroup is used across several spawns (especially if those are executed as persistent workers thousands of times), it increases the size of execution_log_json_file.
We run into this when we use rules_js with:
These node_modules collectively can be hundreds of thousands of files, which multipled with thousands of persistent worker invocations runs easily into millions of lines in execution_log_json_file with the same repeating inputs.
One simple way to address this is to use filegroup if a filegroup appears only once in the execution_log:
Which category does this issue belong to?
Performance
What underlying problem are you trying to solve with this feature?
Reduce the size of execution_log_json_file in cases where several filegroups are used.
Which operating system are you running Bazel on?
MacOS
What is the output of
bazel info release
?release 6.1.0
If
bazel info release
returnsdevelopment version
or(@non-git)
, tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD
?No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
The text was updated successfully, but these errors were encountered: