Poetry 1.1.x breaks mypy in a project with source dependencies #3094

kkozmic · 2020-10-06T04:57:31Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: MacOS 10.16.5
Poetry version: 1.1.1
Link of a Gist with the contents of your pyproject.toml file: link

Issue

This is an issue with mypy under .venv created with latest Poetry (v1.1.x) which works just fine if the .venv was created with older version (v1.0.10 or before)

Our project has two source dependencies on other internal projects.
After the upgrade (and re-creating the .venv) running poetry run mypy . produces a whole bunch of errors like:

error: Cannot find implementation or library stub for module named 'our_source_project1.something'
error: Cannot find implementation or library stub for module named 'our_source_project2.something_else'

Digging through the logs, diffs and then mypy source code I narrowed it down to this:

Under Poetry v1.0 a .venv/lib/python3.8/site-packages/easy-install.pth file gets created which contains paths to each of our code dependency projects.

Under Poetry v1.1 this file doesn't get created.

Thing is, mypy reads this specific file to find these source dependencies. (source).
Without it it doesn't know how to find them, hence the errors, and a broken build.

The text was updated successfully, but these errors were encountered:

finswimmer · 2020-10-06T05:32:51Z

Hello @kkozmic,

is this related: #2194 (comment) ?

fin swimmer

kkozmic · 2020-10-06T05:43:07Z

hey @finswimmer

I looked at that workaround, but if I understand it correctly, mypy_path/MYPYPATH adds other paths for mypy to scan. It's saying "my project's code is in **/*.py but also in this other folder and that second other folder as well".

That's not what I want - I want those additional dependencies to be treated as dependencies, not as my project code.

Additionally, running mypy with mypy_path added fails as it ignores these projects' own mypy settings. Trying to make my project's settings accommodate those dependencies (and keeping them up to date over time) is a recipe for a headache.

I want paths to those two projects to end up in package_path, not mypy_path, like they do under Poetry v1.0.x (and sadly there's no equivalent package_path config value as far as I'm aware).

abn · 2020-10-06T07:41:18Z

@kkozmic I am guessing in this scenario your dependencies are poetry projects installed with develop = true? If so, they should be still written to package-name.pth instead of easy-install.pth

abn · 2020-10-06T07:42:15Z

~~Just read your pypt, this is actually #3086.~~

kkozmic · 2020-10-06T07:54:27Z

Just read your pypt, this is actually #3086.

OK? I saw that issue before submitting this one, and it looked related, albeit concerned with race conditions whereas my problem is broken mypy.

I am guessing in this scenario your dependencies are poetry projects installed with develop = true?

We've got tag = something and branch = something. Not sure how that affects things.

kkozmic · 2020-10-06T08:00:09Z

I tried it with poetry config experimental.new-installer false --local and that doesn't seem to affect this particular problem. Only rolling back to v1.0.10 fixed it

abn · 2020-10-06T08:53:02Z

@kkozmic I was on my mobile. So looking at the mypy code; it seems they should really be loading all pth files as easy-install.pth is only guaranteed by setuptool installs and as a consequence pip. In a PEP 517 world; this is a broken assumption I feel.

EDIT: The code above the referenced one seems to be processing site packages. I wonder why the paths do not get loaded.

EDIT2: Would be great to get a sample project to test this.

kkozmic · 2020-10-06T10:07:38Z

Right, so you're basically saying it's more of a mypy bug than Poetry (which is fair enough).

EDIT2: Would be great to get a sample project to test this.

I'd have to put something together from scratch as obviously I can't use our production code. I'll look into it over the coming weekend

abn · 2020-10-06T10:26:55Z

Right, so you're basically saying it's more of a mypy bug than Poetry (which is fair enough).

Yes. However, if there is anything we can do to keep things compatible, I am happy to look into it. We can, at least for 1.1 play nice and write to easy-install.pth as well if there are no undesired side-effects.

I'd have to put something together from scratch as obviously I can't use our production code. I'll look into it over the coming weekend

Awesome. That would make things easier to triage/decide.

kkozmic · 2020-10-06T10:35:46Z

Yes. However, if there is anything we can do to keep things compatible, I am happy to look into it

Awesome, and I appreciate it 👏

kkozmic · 2020-10-06T11:40:34Z

So I got carried away and created a reproduction @abn

git clone git@github.com:bug-repros/poetry-main.git
cd poetry-main
poetry install
make

This works under Poetry 1.0.10 and fails under 1.1.1 with the error from OP.

bryanforbes · 2020-10-09T20:55:47Z

The next release of mypy (0.800) will include a fix for this, but I'm not sure when it will come out since 0.790 was just released today

kkozmic · 2020-10-12T06:49:10Z

No worries, I'll be keeping an eye out for it. Thanks @bryanforbes

abn · 2020-10-12T13:23:37Z

@kkozmic thank you for the reproducer. It is extremely helpful.

Had a quick look into this, and I am leaning towards not making any changes at the moment as I could not find any other cases where the easy-install.pth is expected/required in a similar fashion. And considering we expect a fix in mypy, adding an additional point of failure seems unwise. The easy-install.pth file is intended to be operated on only by setuptools and with that in mind, poetry writing to it and removing lines from it might end up creating more problems than it solves.

If anyone has a compelling case other than mypy that will also beneifit from this, please do list it here.

bryanforbes · 2020-10-12T14:59:31Z

My only suggestion would be to add a flag to optionally generate easy-install.pth. Right now, I'm using a script to generate easy-install.pth and I have to add it to a few of my projects and remember to run it. I understand, though, if you decide against it.

abn · 2020-10-12T15:28:21Z

To be completely honest, it seems like too much effort for a temporary workaround. I'll think on it.

If there are other cases that are important relying on the file, it might be a different story. Right now, I just haven't found any.

kkozmic · 2020-10-13T01:18:15Z

Fair call @abn . Now 🤞 mypy 0.800 gets released soon so I can upgrade :partyparrot:

edmondop · 2021-01-04T09:43:57Z

Can the problem I have be related to this issue https://stackoverflow.com/questions/65512870/mypy-failing-on-test-source-folders-because-of-absence-of-type-hints-or-library?noredirect=1#comment115833764_65512870 ? And if that's the same cause, is there a known workaround?

andrewtemme · 2021-01-22T20:33:25Z

I just found this issue today when I started running into the problem. Looks like MyPy v.0.800 was released today, 22 Jan 2021. I'm getting this error with the new 0.800 version.

This upgrades `mypy` to version `0.812`. This is primarily motivated by a need to work around python-poetry/poetry#3094, which currently happens if you try to include the SAW Python bindings in a `poetry` project as a `develop` dependency. For the most part, this is a matter of adjusting version bounds. In a handful of cases, we are able to remove `type: ignore` comments due to `mypy-0.8` being smarter than `mypy-0.7`.

dimbleby · 2022-07-31T18:00:07Z

works fine with latest poetry / mypy - dunno which is fixed but this seems good now

github-actions · 2024-02-29T23:06:30Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

kkozmic added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Oct 6, 2020

jonapich mentioned this issue Oct 24, 2020

Relative path cannot be found #3275

Closed

3 tasks

RyanGlScott mentioned this issue Mar 17, 2021

cryptol-remote-api: Upgrade to mypy-0.812 GaloisInc/cryptol#1119

Merged

jorants mentioned this issue Jun 16, 2021

Remove lock file BeanieODM/beanie#58

Merged

fschuch mentioned this issue Jul 28, 2022

mypy: Found module but no type hints or library stubs fschuch/file_storehouse#13

Closed

finswimmer closed this as completed Nov 27, 2022

github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poetry 1.1.x breaks mypy in a project with source dependencies #3094

Poetry 1.1.x breaks mypy in a project with source dependencies #3094

kkozmic commented Oct 6, 2020

finswimmer commented Oct 6, 2020

kkozmic commented Oct 6, 2020 •

edited

Loading

abn commented Oct 6, 2020

abn commented Oct 6, 2020 •

edited

Loading

kkozmic commented Oct 6, 2020

kkozmic commented Oct 6, 2020

abn commented Oct 6, 2020 •

edited

Loading

kkozmic commented Oct 6, 2020 •

edited

Loading

abn commented Oct 6, 2020

kkozmic commented Oct 6, 2020

kkozmic commented Oct 6, 2020 •

edited

Loading

bryanforbes commented Oct 9, 2020

kkozmic commented Oct 12, 2020

abn commented Oct 12, 2020

bryanforbes commented Oct 12, 2020

abn commented Oct 12, 2020

kkozmic commented Oct 13, 2020

edmondop commented Jan 4, 2021 •

edited

Loading

andrewtemme commented Jan 22, 2021

dimbleby commented Jul 31, 2022

github-actions bot commented Feb 29, 2024

Poetry 1.1.x breaks mypy in a project with source dependencies #3094

Poetry 1.1.x breaks mypy in a project with source dependencies #3094

Comments

kkozmic commented Oct 6, 2020

Issue

finswimmer commented Oct 6, 2020

kkozmic commented Oct 6, 2020 • edited Loading

abn commented Oct 6, 2020

abn commented Oct 6, 2020 • edited Loading

kkozmic commented Oct 6, 2020

kkozmic commented Oct 6, 2020

abn commented Oct 6, 2020 • edited Loading

kkozmic commented Oct 6, 2020 • edited Loading

abn commented Oct 6, 2020

kkozmic commented Oct 6, 2020

kkozmic commented Oct 6, 2020 • edited Loading

bryanforbes commented Oct 9, 2020

kkozmic commented Oct 12, 2020

abn commented Oct 12, 2020

bryanforbes commented Oct 12, 2020

abn commented Oct 12, 2020

kkozmic commented Oct 13, 2020

edmondop commented Jan 4, 2021 • edited Loading

andrewtemme commented Jan 22, 2021

dimbleby commented Jul 31, 2022

github-actions bot commented Feb 29, 2024

kkozmic commented Oct 6, 2020 •

edited

Loading

abn commented Oct 6, 2020 •

edited

Loading

abn commented Oct 6, 2020 •

edited

Loading

kkozmic commented Oct 6, 2020 •

edited

Loading

kkozmic commented Oct 6, 2020 •

edited

Loading

edmondop commented Jan 4, 2021 •

edited

Loading