Remote Worker failures on bazel build

[ola-rozenfeld]

When trying to build Bazel using the RemoteWorker, I get intermittent errors such as:

ERROR: /usr/local/google/home/olaola/full-bazel/third_party/BUILD:75:1: Extracting interface //third_party:android_common_25_0_0 failed: ijar failed: error executing command
(cd /usr/local/google/home/olaola/.cache/bazel/_bazel_olaola/098d19699a9b31c0ee0da4e8c41e90b9/execroot/io_bazel &&
exec env -
PATH=/usr/local/google/home/olaola/google-cloud-sdk/bin:/usr/local/google/home/olaola/work/bin:/usr/local/google/home/olaola/depot_tools:/usr/local/google/home/olaola/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/google/home/olaola/google-cloud-sdk/bin:/usr/lib/google-golang/bin:/usr/local/buildtools/java/jdk/bin:/usr/local/sbin
external/bazel_tools/tools/jdk/ijar/ijar third_party/android_common/com.android_annotations_25.0.0.jar bazel-out/local-fastbuild/genfiles/third_party/_ijar/android_common_25_0_0/third_party/android_common/com.android_annotations_25.0.0-ijar.jar): Exit -1.
ERROR: /usr/local/google/home/olaola/full-bazel/third_party/BUILD:75:1: output 'third_party/_ijar/android_common_25_0_0/third_party/android_common/com.android.tools.lint_lint-api_25.0.0-ijar.jar' was not created.

The command actually failed on the RemoteWorker, this is not a gRPC issue.

[ola-rozenfeld]

Another manifestation of the same error (I think):

ERROR: /usr/local/google/home/olaola/.cache/bazel/_bazel_olaola/098d19699a9b31c0ee0da4e8c41e90b9/external/com_google_protobuf_java/BUILD.bazel:1:1: Extracting interface @com_google_protobuf_java//:protobuf failed: ijar failed: error executing command
(cd /usr/local/google/home/olaola/.cache/bazel/_bazel_olaola/098d19699a9b31c0ee0da4e8c41e90b9/execroot/io_bazel &&
exec env -
PATH=/usr/local/google/home/olaola/google-cloud-sdk/bin:/usr/local/google/home/olaola/work/bin:/usr/local/google/home/olaola/depot_tools:/usr/local/google/home/olaola/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/google/home/olaola/google-cloud-sdk/bin:/usr/lib/google-golang/bin:/usr/local/buildtools/java/jdk/bin:/usr/local/sbin
external/bazel_tools/tools/jdk/ijar/ijar external/com_google_protobuf_java/libprotobuf_java.jar bazel-out/host/genfiles/external/com_google_protobuf_java/_ijar/protobuf/external/com_google_protobuf_java/libprotobuf_java-ijar.jar): Exit -1.

[ulfjack]

No idea. There are some cases where we still swallow exceptions from the underlying libraries, and I suspect it's one of those. I'll try to see if I can reproduce this with a patched binary that produces more debugging output.

[ulfjack]

I got a stack trace.

java.io.IOException: Cannot run program "/tmp/remote.pTK0EkckA/build-0121c395-7267-4941-abb9-405621684cdf/external/bazel_tools/tools/jdk/ijar/ijar" (in directory "/tmp/remote.pTK0EkckA/build-0121c395-7267-4941-abb9-405621684cdf"): error=26, Text file busy
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at com.google.devtools.build.lib.shell.JavaSubprocessFactory.create(JavaSubprocessFactory.java:116)
	at com.google.devtools.build.lib.shell.SubprocessBuilder.start(SubprocessBuilder.java:194)
	at com.google.devtools.build.lib.shell.Command.startProcess(Command.java:738)
	at com.google.devtools.build.lib.shell.Command.doExecute(Command.java:705)
	at com.google.devtools.build.lib.shell.Command.execute(Command.java:418)
	at com.google.devtools.build.remote.WatcherServer.execute(WatcherServer.java:250)
	at com.google.devtools.build.remote.WatcherServer.watch(WatcherServer.java:318)
	at com.google.watcher.v1.WatcherGrpc$MethodHandlers.invoke(WatcherGrpc.java:214)
	at io.grpc.stub.ServerCalls$1$1.onHalfClose(ServerCalls.java:148)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:262)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$2.runInContext(ServerImpl.java:572)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=26, Text file busy
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 16 more

[ulfjack]

Well, it's pretty obvious what's going wrong now. Every time we're running an action we download all the relevant inputs and put them all into the same directory, silently overwriting any existing files. If we run two actions in parallel, we can be writing to a file just as we want to execute it, and Linux complains about that.