Disk cache issues on Windows host #28408
Description
Description of the bug:
When using disk cache + remote cache on Windows host, we see the following errors, which seem to also correspond to lost inputs + build restarts. We only observe this on Windows, not Linux or Macos, and this went away when disabling the disk cache.
[6,845 / 8,542] Compiling Rust rlib string (1 file); 0s disk-cache, remote-cache ... (122 actions, 0 running)
WARNING: Remote Cache: D:\a\.cache\bazel-disk-cache\tmp\c7ee7447-cab0-4a7f-9161-b4298e50785e -> D:\a\.cache\bazel-disk-cache\cas\0f\0fcad9be6f325c8f8c34b4eb6188338db56f93b61b1c9e870ff4db0d70f63d31 (Permission denied)
com.google.devtools.build.lib.remote.common.BulkTransferException: D:\a\.cache\bazel-disk-cache\tmp\c7ee7447-cab0-4a7f-9161-b4298e50785e -> D:\a\.cache\bazel-disk-cache\cas\0f\0fcad9be6f325c8f8c34b4eb6188338db56f93b61b1c9e870ff4db0d70f63d31 (Permission denied)
at com.google.devtools.build.lib.remote.util.Utils.waitForBulkTransfer(Utils.java:602)
at com.google.devtools.build.lib.remote.RemoteExecutionService.parseActionResultMetadata(RemoteExecutionService.java:1132)
at com.google.devtools.build.lib.remote.RemoteExecutionService$RemoteActionResult.getOrParseActionResultMetadata(RemoteExecutionService.java:653)
at com.google.devtools.build.lib.remote.RemoteExecutionService.downloadOutputs(RemoteExecutionService.java:1226)
at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:159)
at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:141)
at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:108)
at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)
at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:282)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1222)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1127)
at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)
at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:653)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:780)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:345)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:202)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:471)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Suppressed: com.google.devtools.build.lib.vfs.FileAccessException: D:\a\.cache\bazel-disk-cache\tmp\c7ee7447-cab0-4a7f-9161-b4298e50785e -> D:\a\.cache\bazel-disk-cache\cas\0f\0fcad9be6f325c8f8c34b4eb6188338db56f93b61b1c9e870ff4db0d70f63d31 (Permission denied)
at com.google.devtools.build.lib.vfs.FileSystem.translateNioToIoException(FileSystem.java:858)
at com.google.devtools.build.lib.vfs.JavaIoFileSystem.renameTo(JavaIoFileSystem.java:312)
at com.google.devtools.build.lib.vfs.Path.renameTo(Path.java:631)
at com.google.devtools.build.lib.remote.disk.DiskCacheClient.captureFile(DiskCacheClient.java:140)
at com.google.devtools.build.lib.remote.CombinedCache.lambda$downloadBlobFromRemote$0(CombinedCache.java:456)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:226)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:126)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:491)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:122)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:491)
at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:260)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:172)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:491)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:122)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:491)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:122)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:560)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:237)
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:213)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:172)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:491)
at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:48)
at com.google.devtools.build.lib.remote.util.RxFutures$2.onSuccess(RxFutures.java:251)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onSuccess(SingleFlatMap.java:112)
at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onSuccess(SingleUsing.java:154)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:68)
at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onSuccess(RxFutures.java:155)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:491)
at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:48)
at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onCompleted(GrpcCacheClient.java:469)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:562)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:565)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:733)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:714)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.AccessDeniedException: D:\a\.cache\bazel-disk-cache\tmp\c7ee7447-cab0-4a7f-9161-b4298e50785e -> D:\a\.cache\bazel-disk-cache\cas\0f\0fcad9be6f325c8f8c34b4eb6188338db56f93b61b1c9e870ff4db0d70f63d31
at java.base/sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at java.base/sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.base/java.nio.file.Files.move(Unknown Source)
at com.google.devtools.build.lib.vfs.JavaIoFileSystem.renameTo(JavaIoFileSystem.java:309)
... 64 more
Which category does this issue belong to?
No response
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
No response
Which operating system are you running Bazel on?
No response
What is the output of bazel info release?
No response
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse HEAD ?
If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response