[Bazel CI] ERROR: I/O exception during sandboxed execution: failure in unix_jni.cc - private/var/tmp/_bazel_buildkite (No such file or directory)

[deepalak56]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/5167#019b58d6-89e5-4d89-9b0a-88b14c6e01d1

Platform: MacOS

Logs:

ERROR: /Users/buildkite/Library/Caches/bazel/_bazel_buildkite/2bd44cc40d2abb589f30f3694a0eae77/external/+async_profiler_repos+async_profiler_macos/BUILD.bazel:4:10: Copying files failed: I/O exception during sandboxed execution: [unix_jni.cc:382] /private/var/tmp/_bazel_buildkite (No such file or directory)
(05:10:02) INFO: Elapsed time: 58.655s, Critical Path: 2.39s
(05:10:02) INFO: 1119 processes: 182 remote cache hit, 937 internal.
(05:10:02) ERROR: Build did NOT complete successfully

Culprit:

Steps:

git clone https://github.com/buchgr/bazel-remote
git reset d6dc63124242e697b6896777db0ca83c103643b5 --hard
export USE_BAZEL_VERSION=3b128e82c52f6dee2eceab623144d75ed4effc11
bazel build --config=my-build --profile=/tmp/tmpvimf37qv/build.profile.gz //src:bazel //src:bazel_jdk_minimal //src:test_repos

CC Greenteam @fweikert

[deepalak56]

This issue is affecting multiple repositories (including bazel and rules_testing) on macOS Buildkite agents.
ERROR: I/O exception during sandboxed execution: [unix_jni.cc:382] /private/var/tmp/_bazel_buildkite (No such file or directory)

Observed in:

Repo: rules_testing (Agent: bk-macOs-intel-4ol3)
Repo: bazel (Agent: macOS)

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/5187#019b8c57-ee36-45a3-9aad-cfc99951e0f6

git reset db007bfee840cebcb6f955b80973ba866de38947 --hard
export USE_BAZEL_VERSION=92d426e8ecdb6e947aa47907a114693b1a06cffd

[jin]

OSS team: is it possible to bisect this issue?

[meteorcloudy]

@bazel-io fork 9.0.0

[meteorcloudy]

Unfortunately, I cannot reproduce this locally, and retry usually succeeds.
https://buildkite.com/bazel/bcr-bazel-compatibility-test/builds/743#019bbd06-363e-4e9f-bbcd-5ecbecd88cad