Poetry install with version 1.1.6 fails with file not found errors

[lucinvitae]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian GNU/Linux 10 (buster), as well as macOS Catalina 10.15.7 (19H2)
• Poetry version: 1.1.6 (although seen on 1.1.5 too)
• Link of a Gist with the contents of your pyproject.toml file: One such project experiencing this issue is https://gist.github.com/lucinvitae/d9faa08f9bf9f151aa2032212443ecda but it appears to affect all of our projects using 1.1.6 regardless of pyproject.toml format.

Issue

Poetry install failures with FileNotFoundError exceptions occur for version 1.1.6 and are occasionally blocking the ability to build/deploy our internal apps. This occurs for multiple of our company's projects that install dependencies with poetry. The issue has been reproduced with poetry 1.1.6 and 1.1.5. It occurs locally outside of docker, locally in docker, and in CI using docker. It is sporadic, not affecting every build every time, and thus difficult to reproduce (requires running builds over and over in parallel). Due to the intermittent nature, this typically hits us in CI.

Workaround: We are able to successfully workaround this poetry install flakiness by using Poetry version 1.0.10. Right now we have pinned poetry at that version and are unable to upgrade without seeing significant CI / CD instability. So, we are blocked from using poetry's latest features at scale.

Example failure error:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/~11-0.9.0.dist-info'
...
#13 ERROR: executor failed running [/bin/sh -c poetry install --no-root]: runc did not terminate sucessfully

Several similar FileNotFoundError tickets exist but are for the poetry run command instead of the poetry install command` and do not seem like the same issue:

• [1.0.0b4] poetry run fails with FileNotFoundError #1567
• poetry run raises FileNotFoundError without any more information (workaround found) #3592

We were able to collect several verbose (-vvv) logs for example intermittent failures, which do not occur when retriggering the build. Logs:

poetry-filenotfounderror-1.log
poetry-filenotfounderror-2.log

The exceptions, such as the one in the code block above, typically feature some filename that appears to be truncated (e.g. No such file or directory: '/usr/local/lib/python3.8/site-packages/~11-0.9.0.dist-info. Another example seen recently when running a local macOS Catalina environment:

• Updating typed-ast (1.4.1 -> 1.4.3): Failed

  EnvCommandError

  Command ['/Users/myuser/w/ai/ai-vision-utils/venv/bin/pip', 'install', '--no-deps', '-U', 'file:///Users/myuser/Library/Caches/pypoetry/artifacts/2a/97/67/6a4e444fede37f31a09025fea88c54d4a40d739c4640e72d4c8d355661/typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl'] errored with the following return code 1, and output:
  Looking in indexes: https://my.internal.pypi.repo/repository/invitae-pypi/simple/, https://pypi.python.org/simple/, http://build.locusdev.net:10001/simple/
  Processing /Users/myuser/Library/Caches/pypoetry/artifacts/2a/97/67/6a4e444fede37f31a09025fea88c54d4a40d739c4640e72d4c8d355661/typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl
  Installing collected packages: typed-ast
    Attempting uninstall: typed-ast
      Found existing installation: typed-ast 1.4.1
      Uninstalling typed-ast-1.4.1:
        Successfully uninstalled typed-ast-1.4.1
  ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/Users/myuser/w/ai/ai-vision-utils/venv/lib/python3.8/site-packages/~lick-7.1.2.dist-info'

Note again, the trimmed filename (~lick-7.1.2.dist-info). Rerunning the command fixes the issue, similar to re-triggering the build in CI when we use poetry 1.1.6, which we don't anymore due to this flaky install issue.

Any help would be really appreciated, because we use 1.1.6 locally for some test projects, and really like it otherwise! 🙏

[lucinvitae]

Posting an update to help others with this issue, as well as maintainers that may attempt to debug this issue.

A member of our team has discovered a better workaround for this issue than downgrading from 1.1.6 to 1.0.10:
https://python-poetry.org/blog/announcing-poetry-1-1-0.html#brand-new-installer

Disabling the new poetry installer by running the following command appears to solve the CI/install flakiness issue for us:

poetry config experimental.new-installer false

Breadcrumb trail, see this comment: #3219 (comment)

[finswimmer]

Hello @lucinvitae,

these trimmed filenames are indeed strange and I believe I saw this earlier in another ticket (couldn't find it yet). You should try removing those folders.

fin swimmer

[lucinvitae]

Hello @lucinvitae,

these trimmed filenames are indeed strange and I believe I saw this earlier in another ticket (couldn't find it yet). You should try removing those folders.

fin swimmer

Hello @finswimmer, thanks for reaching out.

These files are created at build time in a workspace which is deleted at the end of the build. So I don't have the ability to delete them afterwards. My guess is they are ephemeral build files created by the new poetry installer, which don't map properly to build requirements at a later stage and cause the resulting build instability. That's just a guess, however.

[abn]

@lucinvitae I think there are several root causes at play here. The ~ prefix file issues iirc was due to how pip installed the wheels from the cache (we should really allow for disabling the cache to debug these issues).

As for this, I suspect it might be due to either a race condition or the resolver picking the wrong package.

[2021-06-04T20:54:49.918Z] #14 238.5   ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensor_float_32_execution.so'

As for this, I suspect this is a network issue or SSL?

[2021-06-04T20:43:15.196Z] #10 26.42   HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/srsly/1.0.5/json (Caused by SSLError(FileNotFoundError(2, 'No such file or directory')))

Would it be possible for you to re-enable parallel builds but with the version from master in your CI environment? We have had a fair few fixes to the new installer. Will be great to get a few things isolated too. It could be that the implementation is just too impatient between steps.

[abn]

As an additional note, I would recommend that you enable virtualenv creation (even in your container). This ensures that compatible support library versions are used (eg: pip/virtualenv) etc. Additionally, I would also recommend poetry installation via pip install --upgrade to also ensure the latest compatible dependencies are installed.

[abdullah-abid-dev]

In my case setuptools was not installed

[neersighted]

In my case setuptools was not installed

There's no way to tell what you had going on without more details -- but from the sound of it, it was far more likely to be #4242. This is related to a weird transient bug with the installer where some race condition is triggered.

[neersighted]

Closing for now due to lack of reproduction and old versions being reported.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.