[EngSys] CI Windows Python3.6 failure: Could not install packages due to an EnvironmentError: [WinError 145] The directory is not empty

[yunhaoling]

CI would sometimes fail because "Could not install packages due to an EnvironmentError: [WinError 145] The directory is not empty" on Windows Python 3.6. Retry could help, but it happens quite frequently.

link to sample error from azure-servicebus CI

...
Successfully built idna-ssl
ERROR: astroid 2.5 has requirement wrapt<1.13,>=1.11, but you'll have wrapt 1.13.3 which is incompatible.
Installing collected packages: urllib3, idna, certifi, charset-normalizer, requests, six, azure-core, pycparser, cffi, cryptography, pywin32, portalocker, PyJWT, msal, msal-extensions, azure-identity, ConfigArgParse, pyyaml, wrapt, multidict, typing-extensions, yarl, vcrpy, azure-devtools, azure-common, azure-mgmt-core, oauthlib, requests-oauthlib, isodate, msrest, azure-mgmt-resource, azure-mgmt-keyvault, webencodings, pyparsing, packaging, bleach, docutils, Pygments, cmarkgfm, readme-renderer, wcwidth, py, attrs, atomicwrites, colorama, zipp, importlib-metadata, more-itertools, pluggy, pytest, pytest-asyncio, json-delta, python-dateutil, azure-storage-common, wheel, coverage, pytest-cov, python-dotenv, azure-mgmt-storage, pyOpenSSL, MarkupSafe, Jinja2, pytoml, azure-sdk-tools, adal, msrestazure, azure-mgmt-servicebus, async-timeout, asynctest, frozenlist, aiosignal, idna-ssl, aiohttp, pip, pytest-custom-exit-code, pytest-forked, execnet, pytest-xdist, smmap, gitdb, GitPython, pbr, stevedore, bandit, chardet, toml, isort, mccabe, typed-ast, lazy-object-proxy, astroid, pylint
  Attempting uninstall: wheel
    Found existing installation: wheel 0.34.2
    Uninstalling wheel-0.34.2:
ERROR: Could not install packages due to an EnvironmentError: [WinError 145] The directory is not empty: 'd:\\a\\_work\\1\\s\\sdk\\servicebus\\azure-servicebus\\.tox\\whl\\lib\\site-packages\\wheel\\'

WARNING: You are using pip version 20.1.1; however, version 21.3.1 is available.
You should consider upgrading via the 'D:\a\_work\1\s\sdk\servicebus\azure-servicebus\.tox\whl\Scripts\python.EXE -m pip install --upgrade pip' command.

=================================== log end ===================================
ERROR: could not install deps [-rdev_requirements.txt, -r../../../eng/test_tools.txt]; v = InvocationError("'D:\\a\\_work\\1\\s\\sdk\\servicebus\\azure-servicebus\\.tox\\whl\\Scripts\\python.EXE'
...

[kristapratico]

related: #22391

[scbedd]

I'm seeing references to possible LongPath conflicts. Given that we randomly generate names, I'm wondering if that's the origin of this problem. I'm going to try enabling LongPath and seeing if these random failures disappear.

I really have no idea where to start on running this down. I've never seen it happen locally. Definitely would welcome any suggestions folks have.

[scbedd]

@kristapratico directed me to pypa/pip#4734 . This is similar, and may be related.

[scbedd]

Going to close this and watch it via build log queries. Will re-open if we see it again.

[scbedd]

Definitely still occurring.

BuildLogLine
| where Timestamp >= now(-20d)
| where Message startswith "ERROR: Could not install packages due to an EnvironmentError: [WinError 145] The directory is not empty:"
| extend IsPipError=Message contains "pip\\\\_internal", IsWheelError=Message contains "wheel\\\\cli", NotEither=Message !contains "wheel\\\\cli" and Message !contains "pip\\\\_internal"
| join kind=leftouter (Build | project BuildNumber, BuildId, DefinitionName) on $left.BuildId == $right.BuildId
| project DefinitionName, BuildId, BuildNumber, Timestamp, Message, IsPipError, IsWheelError, NotEither
| sort by BuildNumber desc
| summarize count() by OccurenceDate=bin(Timestamp, 1d)
| sort by OccurenceDate desc

Investigating other options.

[scbedd]

In the past 20 days.

Need to continue to reduce disk pressure. We've made progress on this issue though.

[scbedd]

Last 50 days.

[scbedd]

This hasn't popped up since April 1st.

We will continue to simplify the installation work being completed during builds. As those improvements filter in, we should see the complete elimination of this error.

Given how sparsely it appears, I'm just going to close this with comment. If it repros we'll re-open.

[kristapratico]

This seems to be showing up in CI again. I've seen it several times, here's a recent example: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2052197&view=logs&j=ea5faa7c-5c30-585e-397e-fb65189f31b0&t=4c9d10f4-ab22-5df2-2620-2ded93d53917&l=255

edit: last 20 days

[scbedd]

😭 😭 😭 😭

[scbedd]

This hasn't occurred in the last 60 days.

DataSource

BuildLogLine
| where Timestamp >= now(-60d)
| where Message startswith "ERROR: Could not install packages due to an EnvironmentError: [WinError 145] The directory is not empty:"
| extend IsPipError=Message has "pip\\\\_internal", IsWheelError=Message has "wheel\\\\cli", NotEither=Message !has "wheel\\\\cli" and Message !has "pip\\\\_internal"
| join kind=leftouter (Build | project BuildNumber, BuildId, DefinitionName) on $left.BuildId == $right.BuildId
| project DefinitionName, BuildId, BuildNumber, Timestamp, Message, IsPipError, IsWheelError, NotEither
| sort by BuildNumber desc
| summarize count() by OccurenceDate=bin(Timestamp, 1d)
| sort by OccurenceDate desc