1.7.0: Cannot find implementation or library stub for module named "scribe"

[mtelka]

While updating mypy package for OpenIndiana from version 1.6.1 to version 1.7.0 I noticed two new test failures:

___________________ testIgnoreImportIfNoPython3StubAvailable ___________________
data: $(BUILD_DIR)/test-data/unit/pythoneval.test:1567:
Failed: Invalid output ($(BUILD_DIR)/test-data/unit/pythoneval.test, line 1567)
----------------------------- Captured stderr call -----------------------------
Expected:
  _testIgnoreImportIfNoPython3StubAvailable.py:4: error: Library stubs not installed for "docutils" (diff)
  _testIgnoreImportIfNoPython3StubAvailable.py:4: note: Hint: "python3 -m pip install types-docutils" (diff)
  _testIgnoreImportIfNoPython3StubAvailable.py:4: note: (or run "mypy --install-types" to install all missing stub packages) (diff)
  _testIgnoreImportIfNoPython3StubAvailable.py:4: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports (diff)
Actual:
  Traceback (most recent call last): (diff)
    File "/tmp/mypy-test-46a1te1g/tmp/_testIgnoreImportIfNoPython3StubAvailable.py", line 2, in <module> (diff)
      import scribe  # No Python 3 stubs available for scribe (diff)
  ModuleNotFoundError: No module named 'scribe' (diff)

Alignment of first line difference:
  E: _testIgnoreImportIfNoPython3StubAvailable.py:4: error: Library stubs not...
  A: Traceback (most recent call last):...
     ^
Update the test output using --update-data -n0 (you can additionally use the -k selector to update only specific tests)
__________________________ testNoPython3StubAvailable __________________________
data: $(BUILD_DIR)/test-data/unit/pythoneval.test:1580:
Failed: Invalid output ($(BUILD_DIR)/test-data/unit/pythoneval.test, line 1580)
----------------------------- Captured stderr call -----------------------------
Expected:
  _testNoPython3StubAvailable.py:1: error: Cannot find implementation or library stub for module named "scribe"
  _testNoPython3StubAvailable.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
  _testNoPython3StubAvailable.py:3: error: Library stubs not installed for "docutils" (diff)
  _testNoPython3StubAvailable.py:3: note: Hint: "python3 -m pip install types-docutils" (diff)
  _testNoPython3StubAvailable.py:3: note: (or run "mypy --install-types" to install all missing stub packages) (diff)
Actual:
  _testNoPython3StubAvailable.py:1: error: Cannot find implementation or library stub for module named "scribe"
  _testNoPython3StubAvailable.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports

Update the test output using --update-data -n0 (you can additionally use the -k selector to update only specific tests)

[mr-c]

We also saw this in Debian; I presumed it was due to the [purposeful] lack of internet connectivity on our build hosts.

https://salsa.debian.org/python-team/packages/mypy/-/commit/d43794178b2f0421654ad0ca2cb0eedd49cde6e0

[mtelka]

I'm pretty confident it is permanent issue, not related to internet connectivity. Proof: testing of 1.6.1 always pass, testing of 1.7.0 always fails.

[mr-c]

@mtelka Could be! We never allow internet connections on our build hosts. I forcibly enabled internet access and testIgnoreImportIfNoPython3StubAvailable still failed so I guess you are right, this is something else.

The second failing test looks like what we experience in Debian with our big package of the entire typeshed: for us, testNoPython3StubAvailable fails because it is missing the extra errors due to the docutils types already being installed locally.

[ilevkivskyi]

I think both failures are caused by docutils being already installed locally. The first one is less obvious, because of how pythoneval tests work: if the type-checking output is empty it tries to actually run the test case using Python interpreter. I am not sure why this is only specific to 1.7, but a random guess is that 1.7 looks for installed packages in more places.

[mtelka]

I just confirmed that when I uninstall types-docutils then both tests pass. I can keep docutils installed; it does not affect those tests. Now there are few questions left:

1. Why both tests passed with 1.6.1 even with types-docutils installed?
2. Is it possible to make both tests to pass again even with types-docutils installed?

[AlexWaygood]

Now there are few questions left:

1. Why both tests passed with 1.6.1 even with types-docutils installed?

The test was changed as part of 10dfafe (see #16226 (comment))

2. Is it possible to make both tests to pass again even with types-docutils installed?

Sure, yes, I just selected a random types-* package that's still actively maintained and isn't installed as part of mypy's CI. There are many such packages; the test could theoretically use any one of these. (That won't fix @mr-c's issue, though -- it'll be impossible for this test to pass if you have all of typeshed's types-* packages installed at once, since the premise of the test is that it's being run in an environment that does not have all currently maintained typeshed packages installed into it.)

Ultimately, I feel the fundamental problem here is that mypy's tests aren't guaranteed to pass if you create an environment that has extra packages etc. installed. I feel like we only guarantee the tests will pass if you create an isolated environment according to the instructions we give in https://github.com/python/mypy/blob/master/CONTRIBUTING.md#getting-started-with-development :)

[mtelka]

2. Is it possible to make both tests to pass again even with types-docutils installed?

Sure, yes, I just selected a random types-* package that's still actively maintained and isn't installed as part of mypy's CI. There are many such packages; the test could theoretically use any one of these. (That won't fix @mr-c's issue, though -- it'll be impossible for this test to pass if you have all of typeshed's types-* packages installed at once, since the premise of the test is that it's being run in an environment that does not have all currently maintained typeshed packages installed into it.)

Why it must be an actively maintained types-* package?

Ultimately, I feel the basic problem is that mypy's tests aren't guaranteed to pass if you create an environment that has extra packages etc. installed. I feel like we only guarantee the tests will pass if you create an isolated environment according to the instructions we give in https://github.com/python/mypy/blob/master/CONTRIBUTING.md#getting-started-with-development :)

Of course, you cannot guarantee interoperability with the whole world. We just need some sort of interoperability with the small subset of that to make our packaging and testing working properly. You do that well. Thanks for that. Some minor issues like this one are just small fee to achieve our goals.

[AlexWaygood]

Why it must be an actively maintained types-* package?

If you don't have stubs for requests installed, and you run mypy over a file that has import requests in it, mypy has a nice feature where it says "Maybe you want to install the stubs for requests?" Mypy generates these suggestions by hardcoding a list of runtime packages for which up-to-date typeshed stubs packages exist. The test we're talking about here checks that this feature is working correctly. The list of suggested stubs need to be hardcoded in mypy, and needs to consist exclusively of typeshed packages (or other packages we have a very high degree of trust in), for security reasons: we don't want to suggest to users that they install untrusted packages from PyPI. (If somebody uses mypy with the --install-types --no-interactive option, moreover, these suggested stubs packages might be automatically installed, heightening the security risk here.)

Okay, let's say typeshed removes its stubs for requests tomorrow. Why might it do that? Here are some hypothetical reasons: It might do that because requests adds inline types, meaning typeshed's stubs are now out of date and redundant. It might do that because there are now stubs packages outside of typeshed that are higher quality than those in typeshed. Or, it might do that because the requests maintainers ask for them to be removed for some reason. If it's any one of these reasons, mypy probably should remove types-requests from the hardcoded list in stubinfo.py: it's probably not a stubs package that it should be suggesting to its users anymore.

The upshot of this all is that, yeah, almost all the stubs packages hardcoded in stubinfo.py are actively maintained typeshed packages. As a result, the stubs package in the test we're discussing here has to also be an actively maintained typeshed stubs package.

[mtelka]

Okay, I think I understand. Thanks for explanation. I was about to suggest typed-ast/types-typed-ast, but maybe you'll remove it from mypy soon.

[AlexWaygood]

I was about to suggest typed-ast/types-typed-ast, but maybe you'll remove it from mypy soon.

yes, we've already removed those stubs from typeshed; we shouldn't be recommending to users that they install the stubs package anymore, as it's no longer maintained. It's arguably a bug that it's still there in the stubinfo.py list in mypy :)

The trouble with a hardcoded list is that it's hard to keep it up to date ;)

[AlexWaygood]

FWIW, some reasons why docutils is a good package for this test from mypy's perspective:

1. It's a popular runtime package that's unlikely to go away anytime soon
2. I think they're unlikely to add inline types themselves anytime soon

[mtelka]

FWIW, some reasons why docutils is a good package for this test from mypy's perspective:

1. It's a popular runtime package that's unlikely to go away anytime soon

Yes, it is so popular that the probability it will be already preinstalled is very high. Otherwise we won't see this bug filled :-).

Then what about something like types-pywin32? Not sure about its popularity or other pros and cons, but at least we will (most likely) never have it installed on OpenIndiana.

[AlexWaygood]

Then what about something like types-pywin32? Not sure about its popularity or other pros and cons, but at least we will (most likely) never have it installed on OpenIndiana.

Sounds fine to me (though I'm not a maintainer here, just a triager and regular contributor :)