Inconsistent --remote_upload_local_results behavior in mixed local and remote execution

[sputt]

Description of the bug:

A remotely executed action cannot use outputs from a locally executed action if they occurred during the same build with --noremote_upload_local_results set. The same build works on retry, since Bazel will upload those outputs as missing inputs.

Background
In our Bazel setup, users are allowed to execute actions remotely, without inputs being uploaded using a CAS only write key to the RBE. User are also able to run actions locally, if requested or if the action does not yet support remote execution. Users are not allowed to write action results to the cache.

In this setup, the first round of executions always works correctly. Bazel uploads any missing inputs to the CAS for remote executions. If an action is run remotely, its outputs end up in the CAS and are found as inputs to any subsequent executions. However, if the first round action ran locally, its outputs are not uploaded and the next remote execution requiring those outputs fails, with the output from the local execution (input file to the remote execution) being missing from the CAS.

Expected behavior
During the first build, the outputs should be uploaded to the CAS in the same way as missing inputs. Action results should still not be written to the cache.

What is the output of bazel info release?

release 7.6.1

Any other information, logs, or outputs that you want to share?

One of our toolchains has a defect where certain rules_foreign_cc libraries do not build correctly remotely, so while we investigate, these actions are marked no-remote-exec. Any downstream library that needs to link against these libraries will fail the first build with an error from the RBE:

Remote Execution Failure:
Not Found: rpc error: code = NotFound desc = Exhausted all peers attempting to read "9a1b0d694d99be1a1b2d7cfc19f0dca9df6f0e31afe3bc28fd18aed7c5cf6879".

[sputt]

Noting this is not C++ specific - any local action followed by a remote execution requiring one of those outputs within the same build will exhibit the issue.

[tjgq]

I tried to repro this with:

genrule(
  name = "a",
  outs = ["a.txt"], 
  cmd = "echo hello > $@",
  tags = ["no-remote-exec"],
)

genrule(
  name = "b",
  srcs = [":a"],
  outs = ["b.txt"],
  cmd = "echo world > $@",
)

but bazel build :b succeeds, so I'm not sure I have the right repro. Can you provide one?

(This also matches my expectation: prior to executing an action remotely we always check for inputs that are missing from the CAS and upload any missing ones; we don't assume that inputs produced by an earlier action have already been uploaded.)

[sputt]

This appears to be fixed in 8.2.1. I was not able to produce the error.

In 7.6.1 it has to do with the size of the input. Here's a repro (at least for our RBE):

load("@bazel_skylib//rules:native_binary.bzl", "native_test")

genrule(
    name = "target",
    outs = ["target.sh"],
    cmd = """echo exit 0 > $@ ; seq 10000000 >> $@""",
    executable = True,
    tags = ["no-remote-exec"],
)

native_test(
    name = "target_test",
    src = ":target",
    out = "target.test",
    args = [
        "arg",
    ],
)

If I try with a smaller file (seq 10000), the test succeeds, but the RBE reports failing one execution attempt on not having an input in the CAS, and a second passing attempt.

This is on Bazel 7.6 with async uploading off. Turning it on does not fix the issue and the profile now looks like this: