Feature request: Generate stub files along during compile

[clbarnes]

Not sure how possible this would be, but it would be awesome if we could give our IDEs an idea of what's going on inside a pyo3 function/class with .pyi stub files. Pyo3 already knows what rust and python types can interconvert: expressing this information for the pyo3 objects in a stub file would really help to bridge the gap.

[pganssle]

I think this is a good idea in general, but to me this feels like actually a big project. Does it need to be in PyO3 itself, or could it be done as a third-party extension?

[clbarnes]

It could certainly be a 3rd-party extension. I'm afraid I don't know enough about pyo3's internals even to judge how big a job it would be - it seemed to me like pyo3 knows which types can interconvert, and rust knows what types its functions support, so it would be a case of formatting existing data rather than trying to generate it in the first place, but I fully accept that that's probably wrong.

[kngwyu]

Sounds interesting, thanks.
But I have to learn .pyi format first 🤔

[konstin]

I've already done a bit of research on the topic, which resulted in #305 and #310.

By default, there are tools that can open native (and pure python) modules and generate stubs from them. Pycharm for example automatically generates stubs for all installed modules; I've added the stub for the word-count example below. Python 3 has additionally introduced __annotations__, which is a dict containing type annotations. However python doesn't allow to set __annotations__ for native functions, so we need to come up with something different. (Another thing to note that's not in the issues linked above is that PEP 563 will change __annotations__ to a Dict[str, str], so that annotations need to be resolved at runtime. However typing.get_type_hints will still work so that shouldn't be a big problem for most tools.)

A simple way to get at least the names and positions of the arguments is generating a text signature (#310), as e.g. orjson currently does manually. I think that that should be done first.

If we want more advanced annotations, we'd need to create our own metadata format. See e.g. wasm-bingen's approach with WasmDescribe::describe(). We need to write the type-info at compile to some constant/field, while writing stub files should be done by an external tool, in our case setuptools-rust or pyo3-pack. wasm-pack has a cli tool and wasm-pack for that

But I have to learn .pyi format first thinking

They are just normal python files with the implementation replaced by ....

# encoding: utf-8
# module word_count.word_count
# from /home/konsti/pyo3/.venv/lib/python3.6/site-packages/word_count/word_count.cpython-36m-x86_64-linux-gnu.so
# by generator 1.147
# no doc
# no imports

# functions

def count_line(*args, **kwargs): # real signature unknown
    """ Count the occurences of needle in line, case insensitive """
    pass

# classes

class WordCounter(object):
    """ Represents a file that can be searched """
    def search(self, *args, **kwargs): # real signature unknown
        """ Searches for the word, parallelized by rayon """
        pass

    def search_sequential(self, *args, **kwargs): # real signature unknown
        """ Searches for a word in a classic sequential fashion """
        pass

    def __init__(self, *args, **kwargs): # real signature unknown
        pass

    @staticmethod # known case of __new__
    def __new__(*args, **kwargs): # real signature unknown
        """ Create and return a new object.  See help(type) for accurate signature. """
        pass


# variables with complex values

__all__ = [
    '__doc__',
    'count_line',
    'WordCounter',
]

__loader__ = None # (!) real value is '<_frozen_importlib_external.ExtensionFileLoader object at 0x7f89093d9940>'

__spec__ = None # (!) real value is "ModuleSpec(name='word_count.word_count', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f89093d9940>, origin='/home/konsti/pyo3/.venv/lib/python3.6/site-packages/word_count/word_count.cpython-36m-x86_64-linux-gnu.so')"

[petr-tik]

hey, thanks for working on this library. I've been looking around about this, so here are my notes, which you might find useful

Looks like mypy solves it by reusing the python parser module to re-parse each module and create an AST. Then it creates Stubs for AST nodes.

https://github.com/python/mypy/blob/master/mypy/stubgen.py

It might be too big a problem to solve inside pyo3, at least mypy shows how one could do it.

[kngwyu]

Yeah but we need here is just type hints for pymethods, right?

[programmerjake]

by manually adding the signatures to the docstrings, pyls can autocomplete all the methods. See #310 (comment) for a proposal

[clbarnes]

Does that include type annotations or is that too much to ask? 😉

My only concern is that if we're writing explicit header stubs anyway, we may as well just write a full pyi stub file and at least gain access to the tooling around that.

[programmerjake]

Does that include type annotations or is that too much to ask?

From what I understand, type annotations don't work at all for return types (CPython doesn't recognize the signature and put it in __text_signature__) and sort-of works for arguments (__text_signature__ gets set properly but inspect.signature raises an exception).

My only concern is that if we're writing explicit header stubs anyway, we may as well just write a full pyi stub file and at least gain access to the tooling around that.

I think having signatures is useful for those cases when you need help() from Python and/or you want inspect.signature to work (don't think .pyi files are used for either, though could be wrong). That should not exclude generating .pyi files in addition, which would also be nice.

[Quogu]

I'd also find this feature really useful.

Ideally what I'd like is that PyO3 produces a .pyi stub with type information both for parameters and return types, and then strips this information out when producing the .pyd file so that the docstring that's included in the .pyd file meets the constraints of the CPython implementation (e.g. no return type annotation). I don't think it's a huge amount of work - tokenising the text_signature string to remove the trailing -> MyType before handling it like we do now, and templating some text to produce the stub. Given the produced stub is just for developer convenience, it doesn't have to be watertight on the first iteration.