Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental parsing issue with namespace packages #12664

Open
PeterJCLaw opened this issue Apr 23, 2022 · 11 comments
Open

Incremental parsing issue with namespace packages #12664

PeterJCLaw opened this issue Apr 23, 2022 · 11 comments
Labels
bug mypy got something wrong topic-incremental

Comments

@PeterJCLaw
Copy link
Contributor

Given a local file which uses something which is a namespace package and then another local file which uses the first one, there's an intermittent bug where the namespace packaged module's types aren't found.
This seems likely similar to #7276, however even with #10937 in place (i.e: on a more recent mypy release) the issue remains.

To Reproduce

$ mkdir demo
$ echo 'import ruamel.yaml' > demo/wrapper.py 
$ echo 'import wrapper' > demo/consumer.py
$ mypy demo  # first run
Success: no issues found in 2 source files
$ mypy demo  # second run
demo/wrapper.py:1: error: Skipping analyzing "ruamel": module is installed, but missing library stubs or py.typed marker
demo/consumer.py:1: error: Skipping analyzing "ruamel": module is installed, but missing library stubs or py.typed marker
demo/consumer.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 2 errors in 2 files (checked 2 source files)

Subsequent runs will alternate between success and failure. Any additional source files in the project (whether they import the wrapper or not) will also have the error message applied to them, which makes failing runs vastly less useful.

Running with --no-incremental appears to avoid the issue; I can run several times in a row with --no-incremental without encountering the failure mode (and have not seen it with that flag enabled).

Running with --namespace-packages appears to have no effect.

Unfortunately as ruamel.yaml is a typed library, ignoring the imports with type: ignore[import] ends up generating more errors as the ignores appear unused on a "successful" run (and you have warn_unused_ignores enabled).

Setup & versions
python3.9 -m venv venv
. venv/bin/activate
pip install -U pip setuptools wheel
pip install mypy ruamel.yaml
$ pip list
Package           Version
----------------- -------
mypy              0.942
mypy-extensions   0.4.3
pip               22.0.4
ruamel.yaml       0.17.21
ruamel.yaml.clib  0.2.6
setuptools        62.1.0
tomli             2.0.1
typing_extensions 4.2.0
wheel             0.37.1

Your Environment

  • Mypy version used: 0.942
  • Mypy command-line flags: see above
  • Mypy configuration options from mypy.ini (and other config files): none
  • Python version used: 3.9 (though similar behaviour observed on other versions; at least on 3.8)
  • Operating system and version: Ubuntu 20.04 LTS
@PeterJCLaw PeterJCLaw added the bug mypy got something wrong label Apr 23, 2022
@hauntsaninja
Copy link
Collaborator

hauntsaninja commented Apr 23, 2022

Thanks, I can repro on master, which means the unreleased #12250 does not fix either

edit: this still repros as of 2022/10/28, so I guess #13124 didn't fix either (looks unrelated to the all_imported_modules_in_file code path). Also linking this comment: #13085 (comment)

@jacleggmsr
Copy link

jacleggmsr commented Nov 21, 2022

I'm getting the same issue.

I'm running python 3.10.7, mypy 0.982, mypy-extensions 0.4.3, ruamel.yaml 0.17.21 and ruamel.yaml.clib 0.2.7 in a larger project.

I followed the exact steps above to repro with a new virtual environment after pip install ruamel-yaml and pip install mypy and fails in the same way.
Output of pip list:

Package           Version
----------------- -------
mypy              0.991
mypy-extensions   0.4.3
pip               22.2.2
ruamel.yaml       0.17.21
ruamel.yaml.clib  0.2.7
setuptools        63.2.0
tomli             2.0.1
typing_extensions 4.4.0

Python 3.10.7, Windows 11.

@sfc-gh-dschumann
Copy link

Hitting the same issue. Do we know what causes this?

@ssbarnea
Copy link
Contributor

ssbarnea commented May 2, 2023

The no incremental does not help with runamel.yaml because the final behavior of mypy depends on presence or absence of the optional binary dependency of it, ruamel-yaml-clib.

When ruamel-yaml-clib is installed, it fails to detect the types for some methods, but in its absence it does. This combined with #8823 puts us in a difficult situation, especially if using strict mode.

@davnat
Copy link

davnat commented May 31, 2023

I can confirm this on Ubuntu 20.04.6 LTS, python 3.8.10, mypy 1.3.0.

In my case the problem appears only in a package that depends on ruamel-yaml very indirectly (it's a dep of a dep of a dep), while in the original package that has the direct dependency and all others in the chain all seems OK, for now.

I noted that the problematic package has a lot of from foo import *, maybe this could trigger the error?

Some more details:

  • in all packages in the chain we're using mypy in strict mode, but in the package where we face the problem we have strict = False in mypy.ini
  • ruamel.yaml.clib is also installed
  • in our case, adding incremental = False in mypy.ini solves the issue... but slows down all builds in CI quite a bit, obviously

@florian-cl
Copy link

I can confirm this issue on MacOS 14.1.2, python 3.11.6, mypy 1.7.1

sbdchd added a commit to recipeyak/recipeyak that referenced this issue Jan 4, 2024
Allow uploading a profile image that gets served / stored by recipeyak

New flow for avatar urls is:

1. if upload is null, generate gravatar
2. if upload is not null, generate link to s3

not strictly related, but the new yaml package is surfacing a bug in
mypy:
python/mypy#12664
@dahlbaek
Copy link

I can confirm this issue on MacOS 13.2.1, python 3.10.10, mypy 1.8.0

@DBCerigo
Copy link

DBCerigo commented May 8, 2024

Similarly still seeing this issue on intermitant mypy runs. --no-incremental fixes it, but also seems to slow down the typechecking significantly (as I think is to be expected), so is far from an ideal solution.

On Ubuntu Debian 22.04 LTS

mypy==1.8.0
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8

@Sh4pe
Copy link

Sh4pe commented Jun 10, 2024

We are affected by this issue as well. Using --no-incremental or simply deleting the .mypy_cache directory each time works as a workaround, but this slows down type checking considerably. Which is obviously not preferred.

We are using Poetry and this are the relevant versions from our poetry.lock:

mypy              1.10.0
ruamel-yaml       0.18.6
ruamel-yaml-clib  0.2.8

I want to say that using mypy is very valuable to us, but this is a small annoyance. Is there a workaround that works with the cache?

@petergaultney
Copy link

python 3.10.13
mypy 1.10.1
google-auth 2.30.0

Likely not relevant, but this does not ever appear to happen with .pth local editable dependencies that are themselves namespace packages. We have lots of those and this issue never occurs with them.

It's also not deterministic, but I can confirm the behavior where it always succeeds when the cache is empty.

@rworkduo
Copy link

rworkduo commented Aug 1, 2024

after a year, this issue is still haunting me and costing me a significant amount of CI running time
is it even being looked into yet?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong topic-incremental
Projects
None yet
Development

No branches or pull requests