In remote execution, when does bazel upload results from the local client?

[JonathanPerry651]

I have a bazel setup with --remote_executor={some address} and with --spawn_strategy=remote, --noremote_upload_local_results, --experimental_dynamic_slow_remote_time=5400s, --remote_timeout=5400s. My remote executor has code which uploads results into its own cache.

I sometimes get a java error stack trace that indicates that bazel is trying to upload results from the local box (specifically, it's a failure in the ByteStreamUploader's uploadBlobAsync code). I don't understand quite how I end up in that codepath - I really don't want to! I have other questions about why it's timing out at all after 600s, but that's a bit secondary to my confusion.

What should my mental model be for how this works please?

Thank you!

[tjgq]

Your setup disables the upload of local action results to the remote cache, but remote execution still requires the inputs of remote actions to be uploaded to the remote cache before they can run. They might be missing from the cache because they're either source files, or outputs of actions that ran locally; in addition, the messages that describe the action to be executed remotely, as well as its input Merkle tree, are also uploaded to the remote cache as blobs.

Are you sure that the blob being uploaded is an output of a local action, and not an input of a remote action? They both go through the code path you mention. Do you have the full stack trace leading up to uploadBlobAsync?

[JonathanPerry651]

Ah that had not occurred to me, thank you.

I am sure of nothing :). I have bazel clean --expunged, and with that command line I would expect nothing to have run locally at all, but maybe I should do some exec logging to see if my expectations are wrong.

I do have the full stack trace, but it's very hard for me to share it unfortunately. Could you let me know what I should look for to confirm or deny please?

Thank you for your quick response!

[JonathanPerry651]

You are exactly right - as confirmed from the exec log, they are source files being uploaded to the cache. Thank you for the pointer!

[JonathanPerry651]

Is there a way to control the chunk size of the upload - i.e. how many files it does at once?

[JonathanPerry651]

To give more details (now that you've helped me understand the situation): I have a dev setup of my remote build execution which for transient reasons has only a single wet piece of string through which to upload source files. In order to fix those reasons, I need to be able to build the world, which includes building the go sdk. We use the really rather cool llvm toolchains (https://github.com/bazel-contrib/toolchains_llvm) code to help us approach hermeticity. This means that building the go sdk involves uploading a metric ton of source files (both by count and by byte-volume) through that wet piece of string, which ends up hitting context deadline exceeded. I've made a very sad workaround, which is to have a trivial little rule that spawns dummy actions consuming 10 sources at a time from the llvm toolchains, and I'm wondering whether there's a better way. Given the transient-ness of this, it's not super important, bnut more understanding is better!

Thank you

[tjgq]

I suggest experimenting with the --remote_max_connections and --remote_timeout flags.

[JonathanPerry651]

ok. thanks again!