Feature request: Filtering execution log

[GabrielGhe]

Description of the problem / feature request:

When using --execution_log_json_file, we generate a very big json file and it's very slow to create this gigantic json file. We're only interested in a couple of points (output files, remote cache hit, size, duration). I know that the duration can be found in the profile log and when we download from the remote cache (created with --profile), but that file doesn't specify exactly what it's trying to download from the remote cache. A single build create an execution log of over 1GB in size so we have to turn it off.

Feature requests: what underlying problem are you trying to solve with this feature?

Add the ability to filter what user wants in the execution log

What operating system are you running Bazel on?

Ubuntu 18.04

What's the output of bazel info release?

release 3.4.1

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

No

[meisterT]

when we download from the remote cache (created with --profile), but that file doesn't specify exactly what it's trying to download from the remote cache

what would you like to see there?

[altonchuzhan]

bazel/src/main/java/com/google/devtools/build/lib/exec/SpawnLogContext.java

Lines 91 to 106 in 09cd1fc

	try {
	for (Map.Entry<PathFragment, ActionInput> e : inputMap.entrySet()) {
	ActionInput input = e.getValue();
	if (input instanceof VirtualActionInput.EmptyActionInput) {
	continue;
	}
	Path inputPath = execRoot.getRelative(input.getExecPathString());
	if (inputPath.isDirectory()) {
	listDirectoryContents(inputPath, builder::addInputs, metadataProvider);
	} else {
	Digest digest = computeDigest(input, null, metadataProvider);
	builder.addInputsBuilder().setPath(input.getExecPathString()).setDigest(digest);
	}
	}
	} catch (IOException e) {
	logger.atWarning().withCause(e).log("Error computing spawn inputs");

I thought it could be done by filter out some big section, for example the input part.
It is quite annoying that I want to fetch a cache hit flag from 8.1G json data in my case.

filter target/action is another option, it may looks like

bazel aquery 'mnemonic("TestRunner", //...)'

[altonchuzhan]

when we download from the remote cache (created with --profile), but that file doesn't specify exactly what it's trying to download from the remote cache

what would you like to see there?

Is there any other way to retrieve a flag fro cache hit on specific target? I got no luck when finding it in profile data.

[GabrielGhe]

@meisterT

[GabrielGhe]

when we download from the remote cache (created with --profile), but that file doesn't specify exactly what it's trying to download from the remote cache

what would you like to see there?

The profile might not be the best place to add extra information about action size/sha256 etc. Ideally we would want some way to filter the execution log or reduce it's size. Our execution log can get to over 5 GB on a clean build.

Open to other ideas. The use case is that we want to be able to find the first actions that were cache misses that invalidated many other actions.

Thanks!