xml package suggests submodules being available ... they're not

[ewerybody]

I was writing up an Issue on the Pylance project as I found a built-ins-example of my seen behavior in question which is the xml package with its stub here:

typeshed/stdlib/xml/__init__.pyi

Line 1 in 72ee95b

import xml.parsers as parsers

So a submodule is imported marked as exported publicly there right away. Now my issue:

I understand that one can import submodules like import xml.parsers and such ... but
If you actually import just xml there is no xml.parsers!

>>> import xml
>>> xml.parsers
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'parsers'

The real file actually hints about 3 more submodules which are also never there on basic xml import.
(Edit: yeah that's intended! __all__ is just about broad imports! Pardon me)

Am I mistaken that the stub and the __all__ declaration suggests that this submodule would be available?
Is this a historic thing to help auto-completers along? I'd think type-checkers are much smarter nowadays.
But maybe I'm just missing something here :)
cheers:
ëRiC

[JelleZijlstra]

Interesting, the real file just has __all__ = ["dom", "parsers", "sax", "etree"]. At runtime, if you do from xml import *, you do get the parsers module out, and xml.parsers also exists. That's a feature of the Python import system I wasn't aware of.

Ideally typeshed should just reflect the runtime behavior, so our stub should have the __all__ and no imports, but I'm not sure if type checkers will like that.

[hauntsaninja]

Yeah, I too learnt about this somewhat recently. mypy does not handle this use of __all__ without a corresponding import to define the symbol, although stubtest has some special handling for the runtime side of things here https://github.com/python/mypy/blob/f1eb04ad379fa7492b88794d2c575461079b478a/mypy/stubtest.py#L167

See also: python/mypy#9935 (comment)

[ewerybody]

AAhhh okok! So __all__ also masks available submodule files in import * ! Thats great!
I was not aware of that (TIL)! I always thought it's about the actually available objects in the regarding scope.

Ok but that would still be covered without ANYTHING in the xml stub, right?
So shouldn't this be empty then?

[JelleZijlstra]

Making it empty would mean that from xml import * doesn't import anything (at runtime; haven't checked type checkers but they ought to behave the same).

[ewerybody]

Aah right! I See! So it works accordingly on the objects within a module
but It also extends to submodules that are not explicitly imported! ☝🤓

OK. I just tested this with having the same line in the .pyi as in the .py: ✔ Works.

I'd suggest to change this so as the current stub implies that you could have xml.parsers if you just import xml. And that's clearly broken.

Making it empty would mean that from xml import * doesn't import anything (at runtime; haven't checked type checkers but they ought to behave the same).

Yeah that's true and off with the real module.
Although: I wonder why this is in place anyway! Why would anyone want to import ALL xml sub packages!?
Seems like from pants put_on all_the_pants where you wouldn't ever need more than one pair of pants 🤷‍♀️

[AlexWaygood]

The following lines in the py3_common stubtest allowlist are there as a result of this weirdness with respect to __all__:

typeshed/tests/stubtest_allowlists/py3_common.txt

Lines 624 to 636 in a82a4bc

	email.base64mime
	email.charset
	email.encoders
	email.errors
	email.feedparser
	email.generator
	email.header
	email.iterators
	email.message
	email.mime
	email.parser
	email.quoprimime
	email.utils

and

typeshed/tests/stubtest_allowlists/py3_common.txt

Lines 706 to 708 in a82a4bc

	xml.dom
	xml.etree
	xml.sax

[AlexWaygood]

distutils.command.__init__ is yet another module that has an __all__ publicly declaring many objects to be available, even though they aren't actually imported (here's what it looks like at runtime).

The typeshed stub for distutils.command.__init__, much like xml with respect to xml.parsers, currently includes __all__ anyway, and pretends that the submodules are imported, even though at runtime they're not.