Detect and fail early on files being modified

[sluongng]

I ran into this issue several times when developing bazel.git

ERROR: .../work/bazelbuild/bazel/BUILD:186:8: 
  Writing: bazel-out/k8-fastbuild/bin/bazel-srcs.tar failed: 
  (Exit 34): Error while uploading artifact with digest 'a38de197744df669165f4c7e570a72a4c0b975dadceb1daee2b8f47e86377a3a/1798950'

The error message was very cryptic so I was unable to tell why it failed.

After weeks of failing on the same error and the same target, I started capturing grpc log and got this

{
  "metadata": {
    "toolDetails": {
      "toolName": "bazel",
      "toolVersion": "8.5.0"
    },
    "actionId": "9911b5f0970d5928414c8402d4459810e0b707a7613e537f3b0cf165d97f6fc3",
    "toolInvocationId": "fa0e9dfa-c2b0-4f34-af04-51b5ecc8f172",
    "correlatedInvocationsId": "2e2b855c-db85-4d79-a8a9-5545e4b5ae3b",
    "actionMnemonic": "PackageTar",
    "targetId": "//:bazel-srcs",
    "configurationId": "be87655a66a18c9ff9b61703f1090ba23fb4f41b0500861ff03594d41f40d445"
  },
  "status": {
    "code": 15,
    "message": "Hash of uploaded bytes \"c04e40b329de02fb98e137e62eb8390c7601ead9ae5c65a0a3d49082d0ec311e\" [BLAKE3] did not match provided digest: \"a38de197744df669165f4c7e570a72a4c0b975dadceb1daee2b8f47e86377a3a\" [BLAKE3]."
  },
  "methodName": "google.bytestream.ByteStream/Write",
  "details": {
    "write": {
      "resourceNames": [
        "uploads/4650315d-8903-4b0b-b726-d4de7047611e/compressed-blobs/zstd/blake3/a38de197744df669165f4c7e570a72a4c0b975dadceb1daee2b8f47e86377a3a/1798950",
        ""
      ],
      "offsets": [
        "0"
      ],
      "finishWrites": [
        "4970260"
      ],
      "numWrites": "304",
      "bytesSent": "4970260"
    }
  },
  "startTime": "2026-01-02T08:39:58.407Z",
  "endTime": "2026-01-02T08:40:00.304Z"
}

This is weird, because (a) I could not find the file anywhere in my source tree and (b), note how the actual uploaded bytes (bytesSent) are 3 times the size of the blob in the digest (50MB vs 18MB). And the bytesSent are supposed to be compressed data too!

So I did a bit of digging and was able to find the Action proto in the remote cache, walked the input root tree digest to find what this file is supposed to be:

  {
    "name": "compact_exec_log.binpb.zst",
    "digest": {
      "hash": "a38de197744df669165f4c7e570a72a4c0b975dadceb1daee2b8f47e86377a3a",
      "sizeBytes": "1798950"
    },
    "isExecutable": true
  }

So I had --execution_log_compact_file=%workspace%/compact_exec_log.binpb.zst set locally to send the log to our BES server. That got caught in this glob of //:srcs

  srcs = glob(
      ["*"],
      exclude = [
          "bazel-*",
          "out",
          "output",
          ".*",
      ],
  ) + [...]

So I fixed it by adding a new exclusion to the glob

---

Anyway, I think there are several asks after this experience:

1. The error message in case of a grpc failure should be more verbose. Users should not have to add remote_grpc_log to see why the ByteStream.write failed.

2. There should be something to detect files still being written to live so that we avoid uploading them and fail early. In this specific case, I think guard_against_concurrent_changes is already enabled in lite mode and does not do that. I suspect we can add a new check that fails when more data is uploaded than was specified in the digest(?)

3. We should have an 'auto' mode that writes execution_log_compact_file to the output_base, similar to Bazel profile. That way, users can still capture the compact exec log and send it to BES without having to manage it inside their workspace or a TMP dir somewhere.