"directory is not empty" when installing a package via pip

[eliot1785]

• Pip version: 9.0.1
• Python version: 3.6
• Operating system: Windows 7

Description:

When I install packages using pip, I am usually (but not always) getting exceptions that occur AFTER the message that the package was successfully installed.

Please note that this is an Anaconda distribution, but I was running pip instead of conda nonetheless. This is on a work computer so I suppose it could be antivirus-related, but I don't have anything specific to suggest that.

Potentially related issue here (#2892) except that the directories in my case were not all that deep.
Another potentially related issue here (#306) but I take it from the fact that the issue was closed that I may have encountered something new.

What I've run:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\sdewey> pip install django
Collecting django
Using cached Django-1.11.5-py2.py3-none-any.whl
Requirement already satisfied: pytz in c:\users\sdewey\appdata\local\continuum\anaconda3\lib\site-packages (from django)

Installing collected packages: django
Successfully installed django-1.11.5
Exception:
Traceback (most recent call last):
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 391, in _rmtree_unsafe
os.rmdir(path)
OSError: [WinError 145] The directory is not empty: 'C:\Users\sdewey\AppData\Local\Temp\pip-build-nr7rei3l\django
\django\contrib\contenttypes\locale\id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip\commands\install.py", line 385, in run
requirement_set.cleanup_files()
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip\utils\build.py", line 38, in exit
self.cleanup()
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip\utils\build.py", line 42, in cleanup
rmtree(self.name)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 49, in wrappe
d_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 212, in call
raise attempt.get()
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip_vendor\six.py", line 686, in reraise
raise value
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip\utils_init_.py", line 102, in rmtree
onerror=rmtree_errorhandler)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 384, in _rmtree_unsafe
rmtree_unsafe(fullname, onerror)
[Previous line repeated 2 more times]
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\shutil.py", line 393, in rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "C:\Users\sdewey\AppData\Local\Continuum\Anaconda3\lib\site-packages\pip\utils_init.py", line 114, in rmtree
errorhandler
func(path)
OSError: [WinError 145] The directory is not empty: 'C:\Users\sdewey\AppData\Local\Temp\pip-build-nr7rei3l\django
\django\contrib\contenttypes\locale\id'

[pradyunsg]

IFAICT, this is a duplicate of #2892 (thus #3055). I'll defer to a core committer to actually decide if this should be closed as a duplicate.

[adj7388]

I got rid of this issue by using "Run as Administrator" when running cmd.exe. Seems weird on Windows, but is it a permissions problem?

[pradyunsg]

So... The only way this can happen is if somehow pip is unable to delete the folder after trying to 5 times over a period of 3 seconds.

The only reason I think this can happen would be something like an Antivirus holding a reference/lock to that folder.

I'm curious what the output of dir <folder-that-wasnt-deleted> is. Could someone try that?

[mhsmith]

The retry decorator applies to the top-level rmtree operation, not to each individual directory in the tree. For large trees, it's possible that the total number of retries exceeds the limit, even if no directory ever required more than 2 attempts.

So perhaps the decorator should be moved to the error handler, and the handler changed so that it always retries, not just when it changed the file mode.

[mhsmith]

And to answer your other question: whenever this happens to me, I find that the directory is always empty by the time I can look at it, and can be removed manually without any problem.

I agree that it's probably interference from anti-virus software. I'm using Windows 7 with Avast, pip 10.0.1.

[shahzadir]

if you are getting issue with nonempty directory problem, just run as administrator privileges and select "just me" in "install for" option.

[cjerdonek]

See also the related issue #6321. PR #6029 looks related, but that occurred while uninstalling rather than installing. In any case, users should see if this is reproducible with pip 19.0.3.

[valorl]

We're also having this issue sporadically, likely due to antivirus (Cylance) inspecting the to-be-removed files. It seems like the retry fix from #2397 does not seem to help here.
Version: pip 19.2.1
I can also confirm that the files in the folder are empty when checked after pip install throws the error.

Here's a process monitor capture for the file/folder in question, where it's visible that Cylance starts to read the file slightly after pip attempts to delete.

[pfmoore]

This sounds like a bug in the antivirus software. If it can't be configured to read files with a mode that allows deletion, then maybe it can be told not to scan pip's temporary directory?

[FreifunkerEZ]

I had the same error updating some modules with pip in a Virtualbox VM through the vboxsf host-guest shared drive. The host (Windows 10) had a virusscanner and the guest (debian) was running the update. I found that the directories complained about were sometimes empty, sometimes not. But it always hit module-directories where the first letter was replaced with a % like %jango. When that happened during the updating of pip itself, pip - and respectively the whole venv - was broken and I deleted it.

Edit: The windows is running Sophos AV-something (corporate stuff).

[kousu]

We're also seeing this when the directory is on NFS: https://forum.spinalcordmri.org/t/installing-sct-4-3-fails-pycache-not-empty/493.

Is there some environment var we can set to make pip more tolerant of slow disks?

[NulAsh]

Recently:

Requirement already satisfied: pip in l:\python39\lib\site-packages (21.1.3)
Collecting pip
  Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 1.7 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
ERROR: Could not install packages due to an OSError: [WinError 145] Папка не пуста: 'L:\\Python39\\Lib\\site-packages\\~ip\\_vendor\\html5lib\\treebuilders'

Sometimes I also see similar error when I delete folder with Far Manager. It tells me "Folder is not empty" and asks what to do with this kind of error, I just select "Skip all", so all files are deleted and all folders where it's possible. In the second step I just delete remaining folders. So maybe this is a solution for recursively deleting folders - first delete what you can, ignoring the "dir not empty" errors, after that if there were such errors, wait 2 seconds and retry the whole procedure.

[NulAsh]

And maybe do 5 retries - but with 2 seconds interval between them. Should be enough.

[DetachHead]

could be your IDE locking files. in my case it worked after closing vscode

[NulAsh]

or antivirus - and this is more frequent

[uranusjr]

I don’t think retries help that much, to be honest, because in my experience the common cases are IDE or Explorer, both won’t unlock the file no matter how many times you try. Instead, we can simply show a warning message telling the user something fails to delete, move the thing to somewhere else (the dash-prefixed scheme we use elsewhere is good enough, or the % prefix mentioned earlier in thread), and tell the user to manually remove it.

Feel free to submit a pull request for this.

[NulAsh]

There's a bug in Windows or maybe caching issues - when you delete all files from folder and them immediately try to delete folder, sometimes it thinks there are still files inside. It have nothing about IDE or explorer, and retries will help.