Add support for reading a list of target patterns for from a file

[robbertvanginkel]

Description of the problem / feature request:

Add support for reading a list of target patterns for from a file for bazel {build,test} (in addition to supporting target patterns in argv)

Feature requests: what underlying problem are you trying to solve with this feature?

We have a command based on query that selects a set of targets (based on this bazel-discuss and #7962) and passes these to bazel build. Roughly something like:

bazel query ... | grep | sed | awk > foo
bazel test $(<foo)

However, this becomes problematic if the output of query + modifications is long, specifically longer than the ARG_MAX the system supports:

# write non-null arg file that is longer than getconf ARG_MAX
$ dd if=/dev/zero bs=1 count=$((1+$(getconf ARG_MAX))) | tr "\0" "\7" > foo
$ bazel build $(<foo)
-bash: /usr/local/bin/bazel: Argument list too long

I currently work around this by using xargs, as it splits the incoming stream into multiple bazel invocations if it is too big.

bazel query ... | grep | sed | awk | xargs bazel test --

I don't however want multiple bazel invocations as this means I need to support reading/merging multiple logs, build event files etc etc.

Ideally bazel would support reading a list of arguments from a file, maybe even generically like:

$ cat argfile.txt
--bazelrc=/dev/null
build
--
//some/...
...
//targets:here
$ bazel --argfile=argfile.txt

But at least I'd like to find a way to have bazel test and build read the list of target patterns from a file (or any other method of passing it a list of arbitrary length).

What operating system are you running Bazel on?

MacOS / Linux

What's the output of bazel info release?

release 0.26.1

Have you found anything relevant by searching the web?

The bazel query documentation suggests using shell command substitution like:

  bazel query deps(//my:target) --output=label | grep ... | sed ... | awk ... > foo
  bazel query "kind(cc_binary, set($(<foo)))"

However, this fails once the query output is longer than ARG_MAX as described above.

bazel/site/docs/query.html

Lines 490 to 493 in aca672f

	<pre>
	bazel query deps(//my:target) --output=label | grep ... | sed ... | awk ... &gt; foo
	bazel query "kind(cc_binary, set($(&lt;foo)))"
	</pre>

• Issue Windows: Add response file support #3750 seems related, but this is not restricted to windows.

[jin]

You can achieve this using:

$ bazel query ... | awk '$0="build:my_target_group "$0' > .bazelrc
$ bazel build --config=my_target_group

Would that work for you?

[kastiglione]

I would like this feature too. Bazel supports actions that use args files, so maybe why not bazel too?

The bazelrc approach would not work for us, our repo has a checked in .bazelrc.

[Globegitter]

@kastiglione you can probably use try-import to have a secondary bazelrc that is on gitignore. Also @robbertvanginkel given the background of this ticket you might be interested in: #8521

[kastiglione]

I didn't know about try-import thanks. I still think bazel should support an args file, it's very common.

[robbertvanginkel]

@jin thanks for the suggestion, I did not know you could put build targets into the bazelrc.

I guess that technically fixes my issue, although I agree with @kastiglione that having general argfile support would be nice. Should I open a general issue for that or do you want to keep this one open and rename it to "Add support for argfiles".

[robbertvanginkel]

@jin I tried this and it works for the number of targets we have in our repository today, but when I tried to do it with a list double the number of targets I get an odd grpc error:

$ bazel query //...:* | awk '$0="build:changed_build_targets "$0' > targetsrc
Loading: 0 packages loaded
$ wc -l targetsrc
   28896 targetsrc
$ bazel --bazelrc=targetsrc build --nobuild --config changed_build_targets
INFO: Invocation ID: 01a66a7e-b60e-443c-be4f-9d097235db10
INFO: Analyzed 28896 targets (45 packages loaded, 41441 targets configured).
INFO: Found 28896 targets...
INFO: Elapsed time: 50.703s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
$ bazel query //...:* | awk '$0="build:changed_build_targets "$0' >> targetsrc # append and see if this scales with twice the number of targets
$ wc -l targetsrc
   57792 targetsrc
Loading: 0 packages loaded
$ bazel --bazelrc=targetsrc build --nobuild --config changed_build_targets

Server terminated abruptly (error code: 1, error message: 'Received RST_STREAM with error code 8', log file: '/private/var/tmp/_bazel_robbert/b21ffbdb27bc569a3409d8e5058938fa/server/jvm.out')

$ cat /private/var/tmp/_bazel_robbert/b21ffbdb27bc569a3409d8e5058938fa/server/jvm.out
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/private/var/tmp/_bazel_robbert/install/1a037b6c0d8096293d1eecfde6528fbd/_embedded_binaries/A-server.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Any idea why this could be?

[robbertvanginkel]

You can achieve this using:

$ bazel query ... | awk '$0="build:my_target_group "$0' > .bazelrc
$ bazel build --config=my_target_group

Would that work for you?

We've been using this for a while and while it works, it makes --announce_rc output bloated as that now outputs all targets too.

It would be great if Bazel could support running from argfiles. Any pointers for where to start?

[robbertvanginkel]

I debugged the Server terminated abruptly (error code: 1, error message: 'Received RST_STREAM with error code 8') error message a bit further, looking at the server's logs I can see the following:

200107 22:26:32.939:WT 18 [io.grpc.netty.NettyServerStream$TransportState.deframeFailed] Exception processing message
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 4211608
	at io.grpc.Status.asRuntimeException(Status.java:523)

grpc/grpc-java's DEFAULT_MAX_MESSAGE_SIZE is 4 MiB so it seems like the earlier proposed trick isn't a sufficient workaround for reading a list of targets from a file. We've encountered some long target lists that this won't work anymore.

I looked a bit through Bazel's source code to find out where to add functionality to read target patterns from a file and stumbled upon the undocumented --experimental_allow_project_files.

bazel/src/main/java/com/google/devtools/build/lib/runtime/CommonCommandOptions.java

Lines 435 to 443 in f17441a

	@Option(
	name = "experimental_allow_project_files",
	defaultValue = "false",
	documentationCategory = OptionDocumentationCategory.UNDOCUMENTED,
	effectTags = {OptionEffectTag.CHANGES_INPUTS},
	metadataTags = {OptionMetadataTag.EXPERIMENTAL, OptionMetadataTag.HIDDEN},
	help = "Enable processing of +<file> parameters."
	)
	public boolean allowProjectFiles;

Based on what I've read in ProjectFile.java and ProjectFileSupport.java it seems like project files should allow a user to specify a build command like

bazel build --experimental_allow_project_files -- +path/to/projectfile

where path/to/projectfile would be a file that contains some target patterns/modifiers. I haven't been able to find the format of this projectfile anywhere, as the code path requires a ProjectFile.Provider but it seems like none is provided to the BlazeRuntime:

bazel/src/main/java/com/google/devtools/build/lib/runtime/BlazeRuntime.java

Lines 1546 to 1554 in f9bc968

	ProjectFile.Provider projectFileProvider = null;
	for (BlazeModule module : blazeModules) {
	ProjectFile.Provider candidate = module.createProjectFileProvider();
	if (candidate != null) {
	Preconditions.checkState(projectFileProvider == null,
	"more than one module defines a project file provider");
	projectFileProvider = candidate;
	}
	}

as no module implements createProjectFileProvider() https://source.bazel.build/search?q=.createProjectFileProvider&sq=&ss=bazel.

Should I try to implement a ProjectFile.Provider and send in a PR? Is there a google internal one that never made it into opensource? Is there a plan to make --experimental_allow_project_files non experimental at some point? I'd really like to have a non-hacky way to have bazel read a list of target patterns to build from a file.

cc @vladmos who committed related to the flag in 58e3891.

[michajlo]

There's some prior art for this in query (--query_file, see also QueryEnvironmentBasedCommand.java). This could be a more direct route with less gotchas than full param file support or "project file" support.