Bugfix: Split large RemoteOutputService gRPC message

[moroten]

Description

Directories with too many output files creates unlimited gRPC messages for the RemoteOutputService. Limit the messages to maxOutboundMessageSize.

Motivation

Large directory outputs from actions in combination with Remote Output Service gives:

ERROR: /code/.../BUILD.bazel:3:16: Action large_directory_ failed: (Exit 34): RESOURCE_EXHAUSTED: grpc: received message larger than max (18681338 vs. 4194304)
Target //:large_directory failed to build

Example rules.bzl:

def _large_directory_impl(ctx):
    output_dir = ctx.actions.declare_directory(ctx.label.name + "_")
    args = ctx.actions.args()
    args.add_all([output_dir], expand_directories = False)

    ctx.actions.run_shell(
        outputs = [output_dir],
        command = """
cd $1
for i in $(seq 1 100); do
    mkdir $i
    for j in $(seq 1 1000); do
        echo $i-$j > $i/$j
    done
done
""",
        arguments = [args],
        execution_requirements = {"supports-path-mapping": "1"},
    )
    return DefaultInfo(files = depset([output_dir]))

large_directory = rule(
    implementation = _large_directory_impl,
)

Example BUILD.bazel:

load("//:rules.bzl", "large_directory")

large_directory(
    name = "large_directory",
)

Build API Changes

No

Checklist

• I have added tests for the new use cases (if any).
• I have updated the documentation (if applicable).

No tests has been added, because no tests of --experimental_remote_output_service exists. The genrule target above pass with this solution.

Release Notes

RELNOTES: None

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[coeuvre]

Thanks for the PR. The code looks good to me, but can we have some unit tests around the split?

[coeuvre]

Please comment why 7 (saying "chose arbitrary" is fine).

[coeuvre]

ditto.