types-protobuf 3.14.19 broken

[ziima]

Since the release of types-protobuf 3.14.19, the type checking based on this package produces a number of errors such as

Library stubs not installed for "google.protobuf.empty_pb2" (or incompatible with Python 3.9)

MVE

#/tmp/test_protobuf.py 
from google.protobuf.empty_pb2 import Empty

e = Empty()

python3 -m venv test-protobuf-venv
source test-protobuf-venv/bin/activate
pip install mypy types-protobuf
mypy /tmp/test_protobuf.py

The script fails with types-protobuf 3.14.19, but works with 3.14.18, regardless of python version (errors observed for 3.7 to 3.10).

[hauntsaninja]

I believe this should be fixed with --namespace-packages (or namespace_packages = True). This will be the default for mypy soon.

[ziima]

@hauntsaninja That applies to types-protobuf itself or to its usage?

[ziima]

Found it.

mypy --namespace-packages /tmp/test_protobuf.py

works fine. That's probably the issue.

[heroesm]

Reproduce in Python3.7 and types-protobuf==3.19.14

python3 -m pip install -U types-protobuf

Requirement already satisfied: types-protobuf in ... (3.18.3)
Collecting types-protobuf
  Downloading types_protobuf-3.19.14-py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 103 kB/s
Installing collected packages: types-protobuf
  Attempting uninstall: types-protobuf
    Found existing installation: types-protobuf 3.18.3
    Uninstalling types-protobuf-3.18.3:
      Successfully uninstalled types-protobuf-3.18.3                                     
Successfully installed types-protobuf-3.19.14

if type mypy ; then \                                                                                                                                                                         
        mypy --install-types --non-interactive ... || mypy ... ; \
fi                                                                                                                                                                                            
mypy is /home/users/.../.pyenv/shims/mypy  
Installing missing stub packages:                                                                                                                                                             
/home/users/.../.pyenv/versions/3.7.5/envs/py37env/bin/python3.7 -m pip install types-protobuf

Requirement already satisfied: types-protobuf in /home/users/.../.pyenv/versions/3.7.5/envs/py37env/lib/python3.7/site-packages (3.19.14)
WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
You should consider upgrading via the '/home/users/.../.pyenv/versions/3.7.5/envs/py37env/bin/python3.7 -m pip install --upgrade pip' command.

nscan/plugin.py:10: error: Library stubs not installed for "google.protobuf.message" (or incompatible with Python 3.7)
nscan/plugin.py:10: note: Hint: "python3 -m pip install types-protobuf"
nscan/plugin.py:10: note: (or run "mypy --install-types" to install all missing stub packages)
nscan/plugin.py:10: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 905 source files)
nscan/plugin.py:10: error: Library stubs not installed for "google.protobuf.message" (or incompatible with Python 3.7)
nscan/plugin.py:10: note: Hint: "python3 -m pip install types-protobuf"
nscan/plugin.py:10: note: (or run "mypy --install-types" to install all missing stub packages)
nscan/plugin.py:10: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 905 source files)

But after downgrade types-protobuf==3.19.13 the issue disappears.

Run mypy with --namespace-packages also works

[aneeshusa]

This is a pretty painful breaking change. I see in #6106 that it was discussed to wait until namespace packages was the default (python/mypy#9636), but that latter PR seems a bit stalled.

At my work we have libraries float their deps to catch issues before they hit applications; since we're not pinning I can't complain about the breakage. However, the fact that there isn't a release of mypy with namespace packages as the default makes this pretty painful, as we're now looking to have to add the namespace_packages configuration to all those repos. protobuf is also a library that by its nature tends to get used widely across a company's repos.

Can we please consider reverting #6106 (merged just 18 hours ago) until there is a new mypy release with namespace packages as the default and an accompanying blog post? For example, the recent switch to decoupled stubs was also a bit painful but having the great docs/blog post made it much smoother. python/mypy#9636 (comment) has some other really great suggestions like improving the error message. @nipunn1313 @JelleZijlstra @hauntsaninja

[srittau]

Reopening to allow discussion of reverting this.

[gurelkaynak]

This is definitely a breaking change IMO. I regularly update minor versions and this is totally unexpected. A minor version update shouldn't make me add flags and change other flags and so on.

Here's what happened:
The minor version update broke our pipeline in an MR. I had to enable namespace_packages but then it started to include some other files that were excluded before because of the follow_imports thing. Then I spent the whole day to try to figure out these things and finally fixed it by enabling namespace_packages and changing follow_imports to "silent".

The documentation doesn't state that even if I excluded some folders it can be included by a follow_imports functionality (or I couldn't find anything about this).