Windows: rebuild and clean sometimes fail with DirectoryNotEmptyException #5907

meteorcloudy · 2018-08-16T09:05:38Z

When I use Bazel 0.15.2 to rebuild Bazel I got this error sometimes:

ERROR: C:/tools/msys64/home/pcloudy/workspace/bazel/src/main/java/com/google/devtools/build/skyframe/BUILD:25:1: Building src/main/java/com/google/devtools/build/skyframe/libskyframe.jar (72 source files) and running annotation processors (AutoCodecProcessor, OptionProcessor) failed (Exit 1): java.exe failed: error executing command
  cd C:/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel
  SET LC_CTYPE=en_US.UTF-8
  external/embedded_jdk/bin/java.exe -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9+181-r4173-1.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/x64_windows-fastbuild/bin/src/main/java/com/google/devtools/build/skyframe/libskyframe.jar-2.params
java.io.IOException: Cannot clean 'bazel-out\x64_windows-fastbuild\bin\src\main\java\com\google\devtools\build\skyframe\_javac\skyframe\libskyframe_classes'
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.cleanupDirectory(SimpleJavaLibraryBuilder.java:91)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.prepareSourceCompilation(SimpleJavaLibraryBuilder.java:53)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:113)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:132)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
        at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Caused by: java.nio.file.DirectoryNotEmptyException: bazel-out\x64_windows-fastbuild\bin\src\main\java\com\google\devtools\build\skyframe\_javac\skyframe\libskyframe_classes\com\google\devtools\build
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
        at java.nio.file.Files.delete(Files.java:1126)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$1.postVisitDirectory(SimpleJavaLibraryBuilder.java:85)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$1.postVisitDirectory(SimpleJavaLibraryBuilder.java:73)
        at java.nio.file.Files.walkFileTree(Files.java:2688)
        at java.nio.file.Files.walkFileTree(Files.java:2742)
        at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.cleanupDirectory(SimpleJavaLibraryBuilder.java:71)
        ... 6 more
Target //src:bazel failed to build
INFO: Elapsed time: 36.284s, Critical Path: 14.20s
INFO: 44 processes: 23 local, 21 worker.
FAILED: Build did NOT complete successfully

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ bazel clean
Starting local Bazel server and connecting to it...
INFO: Starting clean.
ERROR: C:/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel/bazel-out/x64_windows-fastbuild/bin/src/main/java/com/google/devtools/build/lib/remote/blobstore/_javac/blobstore/libblobstore_classes/com/google (Directory not empty)

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ ls /c/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel/bazel-out/x64_windows-fastbuild/bin/src/main/java/com/google/devtools/build/lib/remote/blobstore/_javac/blobstore/libblobstore_classes/com/google

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ bazel clean
INFO: Starting clean.
ERROR: C:/users/pcloudy/_bazel_pcloudy/3uxo2r6v/execroot/io_bazel/bazel-out/x64_windows-fastbuild (Directory not empty)

pcloudy@pcloudy0-w MSYS ~/workspace/bazel
$ bazel clean
INFO: Starting clean.

FYI @laszlocsomor

The text was updated successfully, but these errors were encountered:

laszlocsomor · 2018-08-16T09:13:00Z

Does bazel clean eventually succeed?

meteorcloudy · 2018-08-16T09:16:55Z

Yes, it took me three times although.

laszlocsomor · 2018-08-16T09:19:48Z

Can you repro reliably?

meteorcloudy · 2018-08-16T09:23:35Z

It happens constantly for me in the following situation:

# Clone bazel repo @ HEAD
$ bazel build //src:bazel
$ git reset 4c9149d --hard
$ bazel build //src:bazel

meteorcloudy · 2018-08-16T09:24:20Z

But I'm using 0.15.2, I'll try if it still happens with 0.16.1

laszlocsomor · 2018-08-17T13:20:49Z

I believe @meteorcloudy is working on this -- reassigning.

…an up directory Related: #5907 RELNOTES: None PiperOrigin-RevId: 209735282

laszlocsomor · 2018-11-19T13:13:56Z

@meteorcloudy , have you seen this error with recent releases?

meteorcloudy · 2018-11-19T13:24:35Z

I haven't seen this after setting my output user root to C:/src/tmp, I guess it has something to do with the Windows Defender. I didn't receive any bug report from users, either. So I think it's fine to close issue.

meteorcloudy · 2019-01-10T10:56:51Z

Reopen since @laszlocsomor and I have both been seeing this issue for a while again, I'll look into it.

`bazel clean` command sometimes fails on Windows with "Directory not empty" error. But when checking the directory after the failure, we don't see any file or directory under it. After debugging, we cannot find any unclosed file handle, either. However, when trying to delete the bazel output directory with `rm -rf" it succeeds everytime. The reason is that Cygwin worked around a Windows issue at this commit: Alexpux/Cygwin@28fa2a7 The problem is, I quote: "" Intensive testing shows that sometimes directories, for which the delete disposition has already been set, and the deleting handle is already closed, can linger in the parent dir for a couple of ms for no apparent reason (Windows Defender or other real-time scanners are suspect). "" In this commit, we use a similar way to retry deleting the directory when its "deleted" sub-dirs are still lingering under it. Related bazelbuild#5907

meteorcloudy · 2019-02-19T12:04:40Z

Related stackoverflow question: https://stackoverflow.com/questions/31606978/odd-behaviour-when-deleting-files-with-files-delete

laszlocsomor · 2019-02-19T12:10:36Z

Related stackoverflow question: https://stackoverflow.com/questions/31606978/odd-behaviour-when-deleting-files-with-files-delete

That was a great read, thanks!

`bazel clean` command sometimes fails on Windows with "Directory not empty" error. But when checking the directory after the failure, we don't see any file or directory under it. After debugging, we cannot find any unclosed file handle, either. However, when trying to delete the bazel output directory with `rm -rf" it succeeds everytime. The reason is that Cygwin worked around a Windows issue at this commit: Alexpux/Cygwin@28fa2a7 The problem is, I quote: "" Intensive testing shows that sometimes directories, for which the delete disposition has already been set, and the deleting handle is already closed, can linger in the parent dir for a couple of ms for no apparent reason (Windows Defender or other real-time scanners are suspect). "" In this commit, we use a similar way to retry deleting the directory when its "deleted" sub-dirs are still lingering under it. Related #5907 Closes #7462. PiperOrigin-RevId: 234777436

laszlocsomor · 2019-02-28T12:59:37Z

@meteorcloudy : Can we close this bug?

laszlocsomor · 2019-03-04T09:20:46Z

ping

meteorcloudy · 2019-03-04T09:50:22Z

@laszlocsomor Sorry for the late response. Not yet, because in the JavaSimpleLibraryBuilder I sometimes still get DirectoryNotEmpty exception while rebuilding. It is not using our JNI code to delete file and directory, instead it uses MoreFiles.deleteRecursively from guava.

Did you ever have the same error?

laszlocsomor · 2019-03-04T09:51:53Z

Thanks for the update!

I don't remember clearly anymore if I've seen it or not.

meteorcloudy · 2019-03-04T10:23:21Z

Basically I can reproduce after about 50 commits change in Bazel repo,
eg.

$ bazel build //src:bazel
$ git reset HEAD~50 --hard
$ bazel build //src:bazel

laszlocsomor · 2019-04-05T09:16:32Z

@meteorcloudy : Any updates? Is P1 still adequate?

meteorcloudy · 2019-04-05T09:38:51Z

I am not working on this recently. I'm not getting any report from users, so I guess P2 should be more appropriate. But I can still reproduce this bug on my local machine, so I'll keep it open.

laszlocsomor · 2019-04-29T09:06:30Z

Hey @meteorcloudy , I'd like to take a look at this. Can you still repro this locally? Could you help me repro?

meteorcloudy · 2019-04-29T09:11:13Z

@laszlocsomor , thanks!
Yes, I can still repo this, basically following #5907 (comment)

meteorcloudy · 2019-04-29T09:12:05Z

Just reset to an old enough commit which will trigger a rebuild for most of Java code.

meteorcloudy · 2019-07-22T08:50:06Z

This issue will be fixed from guava side, there's internal work going on, so I'm closing this one.

…ean up directory Related: bazelbuild/bazel#5907 RELNOTES: None PiperOrigin-RevId: 209735282

meteorcloudy added type: bug P1 I'll work on this now. (Assignee required) platform: windows area-Windows Windows-specific issues and feature requests labels Aug 16, 2018

laszlocsomor self-assigned this Aug 16, 2018

laszlocsomor assigned meteorcloudy and unassigned laszlocsomor Aug 17, 2018

bazel-io pushed a commit that referenced this issue Aug 22, 2018

SimpleJavaLibraryBuilder.java: Use MoreFiles.deleteRecursively to cle…

624e500

…an up directory Related: #5907 RELNOTES: None PiperOrigin-RevId: 209735282

meteorcloudy closed this as completed Nov 19, 2018

meteorcloudy reopened this Jan 10, 2019

meteorcloudy mentioned this issue Feb 19, 2019

Windows: Fix deleting directory in JNI code #7462

Closed

laszlocsomor added the bazel 1.0 label Feb 28, 2019

meteorcloudy added P2 We'll consider working on this in future. (Assignee optional) and removed P1 I'll work on this now. (Assignee required) labels Apr 5, 2019

laszlocsomor mentioned this issue Jun 3, 2019

Source forest creation fails with inconsistently-capitalised path references #1583

Closed

meteorcloudy closed this as completed Jul 22, 2019

philwo added the team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website label Jun 15, 2020

guibou mentioned this issue Feb 13, 2021

Bazel clean sometimes fails with Directory not empty #13017

Open

luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022

SimpleJavaLibraryBuilder.java: Use MoreFiles.deleteRecursively to cl…

89bf02d

…ean up directory Related: bazelbuild/bazel#5907 RELNOTES: None PiperOrigin-RevId: 209735282

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows: rebuild and clean sometimes fail with DirectoryNotEmptyException #5907

Windows: rebuild and clean sometimes fail with DirectoryNotEmptyException #5907

meteorcloudy commented Aug 16, 2018

laszlocsomor commented Aug 16, 2018

meteorcloudy commented Aug 16, 2018

laszlocsomor commented Aug 16, 2018

meteorcloudy commented Aug 16, 2018

meteorcloudy commented Aug 16, 2018

laszlocsomor commented Aug 17, 2018

laszlocsomor commented Nov 19, 2018

meteorcloudy commented Nov 19, 2018 •

edited

Loading

meteorcloudy commented Jan 10, 2019 •

edited

Loading

meteorcloudy commented Feb 19, 2019

laszlocsomor commented Feb 19, 2019

laszlocsomor commented Feb 28, 2019

laszlocsomor commented Mar 4, 2019

meteorcloudy commented Mar 4, 2019

laszlocsomor commented Mar 4, 2019 •

edited

Loading

meteorcloudy commented Mar 4, 2019

laszlocsomor commented Apr 5, 2019

meteorcloudy commented Apr 5, 2019

laszlocsomor commented Apr 29, 2019

meteorcloudy commented Apr 29, 2019

meteorcloudy commented Apr 29, 2019 •

edited

Loading

meteorcloudy commented Jul 22, 2019

Windows: rebuild and clean sometimes fail with DirectoryNotEmptyException #5907

Windows: rebuild and clean sometimes fail with DirectoryNotEmptyException #5907

Comments

meteorcloudy commented Aug 16, 2018

laszlocsomor commented Aug 16, 2018

meteorcloudy commented Aug 16, 2018

laszlocsomor commented Aug 16, 2018

meteorcloudy commented Aug 16, 2018

meteorcloudy commented Aug 16, 2018

laszlocsomor commented Aug 17, 2018

laszlocsomor commented Nov 19, 2018

meteorcloudy commented Nov 19, 2018 • edited Loading

meteorcloudy commented Jan 10, 2019 • edited Loading

meteorcloudy commented Feb 19, 2019

laszlocsomor commented Feb 19, 2019

laszlocsomor commented Feb 28, 2019

laszlocsomor commented Mar 4, 2019

meteorcloudy commented Mar 4, 2019

laszlocsomor commented Mar 4, 2019 • edited Loading

meteorcloudy commented Mar 4, 2019

laszlocsomor commented Apr 5, 2019

meteorcloudy commented Apr 5, 2019

laszlocsomor commented Apr 29, 2019

meteorcloudy commented Apr 29, 2019

meteorcloudy commented Apr 29, 2019 • edited Loading

meteorcloudy commented Jul 22, 2019

meteorcloudy commented Nov 19, 2018 •

edited

Loading

meteorcloudy commented Jan 10, 2019 •

edited

Loading

laszlocsomor commented Mar 4, 2019 •

edited

Loading

meteorcloudy commented Apr 29, 2019 •

edited

Loading