Windows support (ENOTEMPTY issues)

[joliss]

I'd like to re-open the discussion about ENOTEMPTY problems on Windows, as previously raised in #25.

Technical background: On Windows, a file can be "busy" when our process or another process still has an open file handle. This can happen unpredictably; e.g. a webserver might have an open connection serving the file, or a virus scanner might be accessing it. When you remove a busy file (with fs.unlinkSync or fs.unlink), unlinkSync will return successfully, but the file sticks around on the file system until the handle is released. As a result, when rimraf removes a file and then tries to rmdir the containing directory, the rmdir operation can fail with ENOTEMPTY. (This is from memory and hearsay, so some details may be wrong!)

This causes random sporadic failures - see e.g. the stack trace reported in broccolijs/broccoli#232.

There is currently a workaround implemented in rimraf (d819b12, fixing #25). However, it is based on retrying repeatedly with setTimeout, and it only works in the asynchronous version. Of course, this seems pretty hackish and potentially unreliable. We're using rimraf a lot in the Broccoli plugin ecosystem, and it makes me worried that we'll have lots of issues on Windows down the line.

So I'd love it if we could find a proper fix for this issue, rather than working around it.

It surely must be possible to find a fix - and this is why I'm opening this issue again: For example, commands like rmdir /s can't possibly be relying on weird workarounds, can they? And Windows Explorer lets you delete directories, without random (flickering) failures stemming from your virus scanner accessing some file.

More detail for people wishing to dig in: this gist by @stefanpenner; this comment in libuv's fs.c; libuv's unlink implementation on Windows. I wonder if libuv's very roundabout unlink implementation (presumably to mimic POSIX) might be a contributor. Could this be fixable in libuv, or could we come up with a platform-specific C implementation for rimraf on Windows that doesn't use libuv?

[isaacs]

Patch very much welcome.

At this point, there's so much garbage working around weird windows semantics, I'd even consider a "fix" that spawns a child process running rmdir /s /q for directories, though that is also not ideal.

[stefanpenner]

I believe this is an invariant of the underlying file system on windows. :s.

A filename isn't released until all file active handles to it have been released. On Unix, the filename is available immediately after the unlink, regardless of open file handles.

But maybe someone more familiar with the windows platform can tell me I'm wrong and how we can fix...

[stefanpenner]

In all honesty, MS OSS team has been pretty helpful for other issues. Maybe they have some ideas on this one. Reaching out...

[denieler]

Hi! It's really big pain for windows users, but not for all.

[kanongil]

While you can't fully remove an open file on windows, you can move it to another location, which could maybe be used as a workaround? Unfortunately though, if I remember correctly, you need to move it before you unlink it.

[stefanpenner]

Interesting....

[joliss]

I wouldn't even be opposed to blocking until all files become unbusy, if that turns out to be necessary. I'd just like the deletion algorithm to be solid - as in, the proper way to do it on Windows - rather than some ad-hoc workaround.

[jabjab]

I seem to have encountered the same issue on OSX in a Phonegap project.

[17:26:41] Error: ENOTEMPTY, rmdir 'platforms/ios/CordovaLib'
    at Error (native)

The contents of the directory were successfully removed, but the directory itself was not.

[jnslxndr]

Any news on this?
I am really having trouble on Windows 7 running a CI server (that needs to build the server stuff on Windows and integrate the ember build afterwards, not my favorite...)

[jnslxndr]

I noticed, Windows 8.1 does not have the issue; just Windows 7.
I followed the Windows guides on ember-cli and also tested with the ember-cli-windows helper tool to deactivate defender and the search index. But, no luck.

[jprichardson]

Also having this issue with fs-extra (uses rimraf) after adding appveyor. See more here: https://ci.appveyor.com/project/jprichardson/node-fs-extra/build/1.0.19/job/dlqb9nnn5cw9c5wj (search for ENOTEMPTY). I'll dig around some more to hopefully help with a solution.

[drourke]

I'm on windows 8.1, changing the tmp folder property to hidden fixed it for me.

Experiencing this issue in Windows 7, as well.

[Bouke]

Experiencing this issue on Windows 7 as well. However setting the hidden property didn't resolve the issue.