Bazel 9.2 hangs indefinitely after a remote-cache ByteStream read times out

[snakethatlovesstaticlibs]

Description of the bug:

AI was used to investigate this ticket, but it was reviewed before filing

Bazel 9.2.0 can hang indefinitely when a gRPC ByteStream remote-cache read times out while loading a file through the experimental remote repository contents cache.

The network layer correctly terminates the read after the configured 60-second deadline, but Bazel's download future does not become terminal. One Skyframe thread consequently remains in an unbounded Future.get() while holding the synchronized RemoteExternalOverlayFileSystem monitor. All other Skyframe threads needing that filesystem block behind it, preventing the build from making further progress.

Without an external watchdog, affected CI jobs remain stuck until the one-hour GitLab timeout.

sequenceDiagram
    participant B as Bazel 9.2
    participant G as Istio ingress gateway
    participant F as Buildbarn frontend
    participant S as Storage shard v3-1

    B->>G: ByteStream/Read
    G->>F: Forward read
    F->>S: Fetch 55.7 MB blob
    S-->>F: Complete blob in 911 ms
    F-->>G: Partial downstream stream
    G-->>B: 17.8 MB delivered over 60 s
    G--xB: response_timeout / HTTP/2 stream terminated
    Note over B: Terminal failure does not reach the overall download future
    B->>B: waitForBulkTransfer calls unbounded Future.get()
    Note over B: getInputStream holds the synchronized filesystem monitor
    B->>B: Other Skyframe evaluators block on the same monitor
    Note over B: Build makes no further progress until watchdog or CI timeout

Loading

Cache and network evidence

The affected request was:

• RPC: /google.bytestream.ByteStream/Read
• Request ID: f6ff7f04-40ba-4b0b-9f52-418546d7cf49
• Client IP: 10.101.103.151
• Start: 2026-08-18T18:27:58.868Z

The storage shard was healthy:

• Storage shard: storage-v3-1
• Storage request ID: bed14f69-e733-4392-8ffa-d94d8fc03cbb
• Blob size served: 55,674,024 bytes
• Storage response time: 911 ms
• Response: HTTP 200 / via_upstream

Delivery toward the runner then stalled:

• Buildbarn frontend: buildbarn-frontend-568cb688cb-pjxpp
• Frontend duration: 59,998 ms
• Bytes sent toward the gateway: 33,629,916
• Termination: max_duration_timeout

At the ingress gateway:

• Gateway duration: 60,009 ms
• Bytes delivered toward the runner: 17,827,425
• Response flags: UT
• Termination: response_timeout

This rules out a stale storage endpoint or slow storage shard for this occurrence. The blob reached the frontend in under one second; the runner-bound stream then timed out after partial delivery.

Bazel evidence

Two thread dumps taken 30 seconds apart showed the same state:

• skyframe-evaluator-40 was blocked in:
  • RemoteExternalOverlayFileSystem$RemoteExternalFileSystem.getInputStream
  • Utils.waitForBulkTransfer
  • Future.get
• The thread held the RemoteExternalFileSystem monitor.
• Approximately 48 other Skyframe evaluator threads were blocked waiting for that same monitor.
• No progress occurred between dumps.

In Bazel 9.2.0, getInputStream() is synchronized and waits for the cache download using waitForBulkTransfer(). That ultimately performs an unbounded Future.get().

Expected behavior

After Envoy terminates the RPC, gRPC should invoke the terminal callback and Bazel should:

1. Complete the current download attempt exceptionally.
2. Retry only within the configured retry limit.
3. Eventually fail the cache read or recover without deadlocking the build.
4. Avoid holding a global filesystem monitor during remote I/O.

The total wait should be bounded by the configured RPC timeout and retry policy.

Actual behavior

The proxy terminated the read at 60 seconds, but the overall Bazel download future remained pending. Because Bazel waited on that future while holding the remote-overlay filesystem monitor, all Skyframe evaluation eventually stopped.

Suspected cause

The immediate cause is an incomplete asynchronous future: either the stream termination was not delivered to Bazel's onError() callback, an exception escaped while processing the callback, or a subsequent retry or channel operation became permanently pending.

The lack of an outer timeout in waitForBulkTransfer() allows that condition to persist indefinitely. The synchronized getInputStream() method amplifies one stuck read into a build-wide deadlock.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 9.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

Environment

• Bazel version: 9.2.0
• Platform: Linux ARM64
• gRPC client: grpc-java-netty/1.71.0
• Relevant settings:
  • --experimental_remote_repo_contents_cache
  • --remote_timeout=60
  • Default --remote_retries=5

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

• bazelbuild/bazel#11782: same remote-cache waitForBulkTransfer -> Future.get() hang. Maintainers diagnosed cases where an unchecked RPC callback exception left a future incomplete.
• bazelbuild/bazel#21777: ByteStream read errors not reaching the expected callback, causing a hang.

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@bazel-io fork 9.3.0