Remote worker execution sometimes fails with exit -1

[ulfjack]

I split this out from #3251. I get this exception:

java.io.IOException: Cannot run program "/tmp/remote.pTK0EkckA/build-0121c395-7267-4941-abb9-405621684cdf/external/bazel_tools/tools/jdk/ijar/ijar" (in directory "/tmp/remote.pTK0EkckA/build-0121c395-7267-4941-abb9-405621684cdf"): error=26, Text file busy
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at com.google.devtools.build.lib.shell.JavaSubprocessFactory.create(JavaSubprocessFactory.java:116)
	at com.google.devtools.build.lib.shell.SubprocessBuilder.start(SubprocessBuilder.java:194)
	at com.google.devtools.build.lib.shell.Command.startProcess(Command.java:738)
	at com.google.devtools.build.lib.shell.Command.doExecute(Command.java:705)
	at com.google.devtools.build.lib.shell.Command.execute(Command.java:418)
	at com.google.devtools.build.remote.WatcherServer.execute(WatcherServer.java:250)
	at com.google.devtools.build.remote.WatcherServer.watch(WatcherServer.java:318)
	at com.google.watcher.v1.WatcherGrpc$MethodHandlers.invoke(WatcherGrpc.java:214)
	at io.grpc.stub.ServerCalls$1$1.onHalfClose(ServerCalls.java:148)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:262)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$2.runInContext(ServerImpl.java:572)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=26, Text file busy
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 16 more

I am highly confident that it's this bug:
https://bugs.openjdk.java.net/browse/JDK-8068370

I.e., the problem is that there's a race when multiple threads try to fork and exec binaries that were recently written to disk. The underlying problem is that Linux has no API to do this reliably (as far as we can tell). Workarounds are either to retry when it happens, to use a helper process to do the forking, or otherwise serialize the forking.

[ulfjack]

If this becomes more pressing, the best we can do is retry the subprocess execution, with the downside that that could hide other errors.

[ola-rozenfeld]

Wait, this is an IOException, not a CommandException or AbnormalTerminationException. That means after my fix is in, it will be converted to INTERNAL status and retried by the client.
Still bad, but probably will eliminate it in practice.

[ulfjack]

This was happening less often because I accidentally made all remote execution single-threaded. :-( Now that we've fixed that, I can see this happening fairly commonly, so we should have a fix for this before we release.

[ulfjack]

Ola says that it still happens even after my change, so reopening this.