Crash on Jenkins

[DoDoENT]

Description of the bug:

I'm using Bazel 8.5.1 and I'm in the process of introducing remote build support (I'm using Buildbarn).

Today, I found this crash in my CI logs:

(14:50:38) FATAL: bazel crashed due to an internal error. Printing stack trace:

java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=@@rules_jvm_external++maven+android_test_deps//:org_jetbrains_annotations, config=BuildConfigurationKey[8b5d464c17ab4491db20dabbedc9edfad8d3ee0483f30ff59582897082547bff]}, actionIndex=0}' (requested by nodes 'ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=@@rules_jvm_external++maven+android_test_deps//:org_jetbrains_annotations, config=BuildConfigurationKey[8b5d464c17ab4491db20dabbedc9edfad8d3ee0483f30ff59582897082547bff]}, actionIndex=1}', 'ArtifactNestedSetKey[2]@34477901', 'ArtifactNestedSetKey[3]@1123689307', 'ArtifactNestedSetKey[3]@2004349441')

	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)

	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)

	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

Caused by: java.lang.NullPointerException: ActionInput not found for filename bazel-out/k8-opt-exec-ST-fc026f303eed/bin/external/rules_jvm_external+/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/libzip.jar in CacheNotFoundException

	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1008)

	at com.google.devtools.build.lib.remote.common.BulkTransferException.getLostInputs(BulkTransferException.java:92)

	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.execLocallyAndUploadOrFail(RemoteSpawnRunner.java:608)

	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:363)

	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:161)

	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:121)

	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)

	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:253)

	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1177)

	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1082)

	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)

	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)

	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:566)

	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:946)

	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:377)

	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:218)

	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)

	... 8 more

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Unfortunately, I don't know how to reproduce it.

My invocation was

bazel test --local_resources=cpu=6 --local_resources=ram=32768 --experimental_cpu_load_scheduling=true --config=ci --config=android --config=release --android_platforms=//platforms/android:armeabi-v7a //:android

From BuildBarn BES, I can also see the list of raw command line flags:

Raw Command Line Options:
--nobuild_runfile_links
--enable_platform_specific_config
--jobs=200
--host_platform=@host_platform//:host
--noexperimental_check_external_repository_files
--experimental_fetch_all_coverage_outputs
--heap_dump_on_oom
--incompatible_default_to_explicit_init_py
--incompatible_modify_execution_info_additive
--incompatible_strict_action_env
--module_mirrors=https://bcr.cloudflaremirrors.com
--noremote_upload_local_results
--repo_env=JAVA_HOME=../bazel_tools/jdk
--reuse_sandbox_directories
--nosandbox_default_allow_network
--show_result=20
--test_output=errors
--incompatible_auto_exec_groups
--incompatible_autoload_externally=
--incompatible_config_setting_private_default_visibility
--incompatible_disable_native_repo_rules
--incompatible_disable_non_executable_java_binary
--incompatible_disable_starlark_host_transitions
--incompatible_repo_env_ignores_action_env
--incompatible_autoload_externally=+@rules_python,+@rules_android
--noincompatible_auto_exec_groups
--disk_cache=~/.bazel-cache/build
--experimental_disk_cache_gc_max_size=20G
--repo_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
--guard_against_concurrent_changes=full
--spawn_strategy=dynamic
--experimental_local_lockfree_output
--remote_executor=grpc://armani.microblink.com:8980
--remote_local_fallback
--bes_backend=grpc://konan.microblink.com:8082
--bes_results_url=https://portal.build.microblink.com/bazel-invocations/
--build_metadata=REPO_URL=https://bitbucket.org/microblink/core-identity
--copt=-g
--strip=never
--verbose_failures
--copt=-fvisibility=hidden
--cxxopt=-fvisibility-inlines-hidden
--java_runtime_version=remotejdk_21
--tool_java_runtime_version=remotejdk_21
--features=asan
--features=ubsan
--test_tag_filters=host,starlark
--local_resources=cpu=6
--local_resources=ram=32768
--experimental_cpu_load_scheduling
--config=ci
--announce_rc
--color=yes
--curses=yes
--grpc_keepalive_time=30s
--lockfile_mode=error
--remote_download_outputs=minimal
--remote_local_fallback
--remote_timeout=3600
--remote_upload_local_results
--show_progress_rate_limit=60
--show_timestamps
--show_progress_rate_limit=10
--show_progress
--test_output=errors
--disk_cache=
--experimental_scale_timeouts=5.0
--http_timeout_scaling=5.0
--spawn_strategy=remote,worker,sandboxed,local
--noremote_local_fallback
--flaky_test_attempts=2
--test_summary=terse
--test_summary=short
--config=android
--test_tag_filters=android
--config=no_sanitizers
--features=-asan
--features=-ubsan
--features=-wasm_asan
--features=-wasm_ubsan
--android_platforms=//platforms/android:arm64-v8a,//platforms/android:armeabi-v7a
--host_platform=@host_platform//:host_for_crossbuild
--extra_toolchains=@androidndk//:all
--features=-android_unwind_tables
--config=release
--config=no_sanitizers
--features=-asan
--features=-ubsan
--features=-wasm_asan
--features=-wasm_ubsan
--compilation_mode=opt
--features=thin_lto
--android_platforms=//platforms/android:armeabi-v7a
Explicit Startup Options:
--host_jvm_args=-DBAZEL_TRACK_SOURCE_DIRECTORIES=1
Effective Startup Options:
--max_idle_secs=10800
--noshutdown_on_low_sys_mem
--connect_timeout_secs=30
--output_user_root=/home/jenkins/.cache/bazel/_bazel_jenkins
--output_base=/home/jenkins/.cache/bazel/_bazel_jenkins/a2f833eabd5a14866932073f644767d5
--failure_detail_out=/home/jenkins/.cache/bazel/_bazel_jenkins/a2f833eabd5a14866932073f644767d5/failure_detail.rawproto
--idle_server_tasks
--write_command_log
--nowatchfs
--nofatal_event_bus_exceptions
--nowindows_enable_symlinks
--noclient_debug
--host_jvm_args=-DBAZEL_TRACK_SOURCE_DIRECTORIES=1

Which operating system are you running Bazel on?

MacOS and Linux, but this crash happened on Linux x86_64

What is the output of bazel info release?

release 8.5.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

This happened in our internal closed-source repository I'm not able to share, unfortunately.

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

While this is a duplicate of #27229, it makes it look like an actual input of the action is missing metadata, which would indicate that the original fix isn't correct. I'll look into this.

[DoDoENT]

Happened again today on my cluster:

(11:17:10) FATAL: bazel crashed due to an internal error. Printing stack trace:

java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=@@gazelle++go_deps+org_golang_google_protobuf//internal/filetype:filetype, config=BuildConfigurationKey[788f131c0b8dbce1d85eeee0ddbee654828e352a623eb5d99c5d7cf6c8dccb2b]}, actionIndex=0}' (requested by nodes 'ArtifactNestedSetKey[12]@769937180', 'ArtifactNestedSetKey[6]@600487581')

	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)

	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)

	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)

	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

Caused by: java.lang.NullPointerException: ActionInput not found for filename bazel-out/k8-opt-exec-ST-b613f8e66fd7/bin/external/rules_go+/stdlib_/pkg/linux_amd64/encoding/base32.a in CacheNotFoundException

	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1008)

	at com.google.devtools.build.lib.remote.common.BulkTransferException.getLostInputs(BulkTransferException.java:92)

	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.execLocallyAndUploadOrFail(RemoteSpawnRunner.java:608)

	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:363)

	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:161)

	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:121)

	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)

	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:253)

	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1177)

	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1082)

	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)

	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)

	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:566)

	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:946)

	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:377)

	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:218)

	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)

	... 8 more

[fmeum]

The missing inputs seem to be runfiles or children of a tree artifact. I still don't see how they could be missing though.

@DoDoENT Would you be able to run a custom Bazel binary with additional debug outputs? I could send you a branch based on 8.5.1 that you could review (I will just add a few print statements) and build yourself.

[DoDoENT]

Would you be able to run a custom Bazel binary with additional debug outputs?

I can try. You'll just have to teach me how 😅 . I'm currently using bazelisk to use v8.5.1 and I plan to migrate to v9 as soon as I get answers about how to use rules_cc in symbolic macro in Bazel 9.

My company has still not transitioned from CMake/Conan/Gradle/Xcode/MSBuild/NPM to Bazel - I'm still building the support, and random crashes on CI look bad 😅 .

[DoDoENT]

Also, as a note: I was working on adding support for Bazel in my company since August and was using bazel-remote and never had this issue. However, after migrating several larger parts of the codebase to a Bazel-based monorepo, cache-only remote support became insufficient, especially during this early stage, while I frequently change Bazel flags and invalidate the cache. For that matter, on Friday last week, I started working on introducing BuildBarn into our company, and this is when these problems started occurring. I certainly know that until the very last build (the last trace I pasted above), I had incorrectly configured BuildBarn storage (too large AC for given CAS and way too small key location map for both), which could have contributed to the issue (although in that case I would get the error FAILED_PRECONDITION: Object not found).

However, the last time it occurred, I believe the BB storage was finally configured correctly, although I'll know that for sure during next week.

[fmeum]

Thanks for providing the context, that can be very helpful when I try to reproduce this myself.

I pushed a commit with debug prints to #28412. You can build and run it like this:

1. Check out my branch https://github.com/fmeum/bazel/tree/28382-debug-build and verify that it's really just debug prints on top of the upstream release-8.5.1 branch.
2. Build a Bazel binary with bazel build -c opt //src:bazel. On Linux this should really only require core tools plus gcc.
3. Either copy that binary to your CI machine and use it directly or, if you can only go through bazelisk, upload it somewhere public and have your CI consume it via USE_BAZEL_VERSION=8.5.1234 BAZELISK_FORMAT_URL=<the URL to the bazel binary> bazelisk .... The version after USE_BAZEL_VERSION needs to be unique each time you update the binary and different from any released Bazel version since Bazelisk caches binaries by version. This is a bit of a hack to easily load a binary from a fixed URL.
4. When the crash happens, you should see a lot more detailed output, including all the input and output file names. Feel free to redact whatever you are not comfortable sharing. You can also send me information via email (in my commits).

[DoDoENT]

It doesn't want to happen now for more than two days of constant builds... 🤷🏻

I guess it was due to the misconfiguration of BuildBarn storage.