automatic type stub (pyi) generation for java classes

[michi42]

I'm working on making a rather big, domain specific java API available through JPype. One of our main issues is the complete lack of type information during static analysis - and therefore no linting or IDE auto-completion.

PEP 484 and 561 support providing type information in separate .pyi "stub" files. The mypy project provides stubgen, a tool to automatically create these from python or C modules.
Since Java is strongly typed, all the information necessary to generate these kind of stub files is in principle available in the class files and can be obtained e.g. through reflection.

I implemented a first version: https://gist.github.com/michi42/2110a8615a0e917a13ec6748c6168735
EDIT: Now a standalone repo: https://gitlab.cern.ch/scripting-tools/stubgenj

The module is supposed to be called from a simple script that sets up the JVM, e.g.

import jpype
import stubgenj

# fire up the JVM with the necessary JARs

stubgenj.generate_java_stubs(["java", "cern.lsa"], "pyi")

This is still very limited, just a starting point that handles the "most common" cases. In particular, it does not support Generics (should be translated to python type vars), Constructors (should be translated to init methods) yet, and the types of constants and enum constants are not yet detected.

[Thrameos]

Interesting. I look forward to some additional information.

Can this be connected to the existing annotation API that we are using? We currently support Jedi auto-completion which allows linting of returns types if the overload is not mixed. I haven't tried with Kite.

Here is my recommendation as an implementation. Just like Java docs we can pick up Java information though the resource API. Therefore, if you run a preprocessor that converts the Java jar file (org-mystuff.jar) into a second jar with the needed annotation org-mystuff-py.jar. Then when I load a class using org.mystuff.MyClass.class I can also fetch the org.mystuff.MyClass.annot file which can then provide stubs.

[michi42]

Hi,

Thanks a lot the quick response!

Indeed, it goes into the same direction as the annotation API (and __docs__) that JPype is already populating at run-time (which is great for interactive applications like IPython or Jupyter), however it provides static information that is pre-generated in once and can then be used without executing any of the code (as it is typically the case for IDEs like PyCharm).

Also, this supports overloads, even when the signatures are completely different and return types depend on the signatures, through the overload decorator: https://docs.python.org/3/library/typing.html#typing.overload - also it should be possible to represent generics and type arguments, although my first version does not handle that. I frankly don't know if the annotation API supports all of these.

As for distribution, I'm afraid it can't be (just) a JAR file, as the whole point behind this is that python IDEs (and possible linting tools) can read the type stubs in a purely static way, without having to run any of our code. Therefore, the type stub "shadow packages" have to be packaged and installed in a way common python IDEs support, typically as an egg/wheel. The structure has to match the Java class hierarchy, for instance (after generating "java.*" and some private APIs):

Note that there are only __init__.py files - because Java does not have the concept of modules, only packages and contained classes.

A few results (in PyCharm):

[Thrameos]

Okay that makes sense. Please keep me posted on any hooks that you require in terms of support.

https://www.jetbrains.com/help/pycharm/stubs.html

I guess the correct solution would be to make a jpype-java-v8, jpype-java-v11, etc. package that gets posted to pypi then? We can then have those packages depend on JPype1 to reduce the need to install both separately. Is that what you envision?

This is very similar to what I am working on for the reverse bridge in which Python generates a package with nothing more than interfaces for each Python class. That allows Java to have an interface with each required method exposed in Python.

[michi42]

Okay that makes sense. Please keep me posted on any hooks that you require in terms of support.

Sure, thanks a lot!

I guess the correct solution would be to make a jpype-java-v8, jpype-java-v11, etc. package that gets posted to pypi then? We can then have those packages depend on JPype1 to reduce the need to install both separately. Is that what you envision?

That would indeed be an option to ease the working with java standard library classes (like https://github.com/python/typeshed does for python standard libraries).
Also, I think it could be useful to ship the stub generator along with JPype to allow users to generate stubs for their own domain-specific Java packages.

One potential (minor) issue with static stub generation is that java.lang.String auto-conversion can be switched on and off at run-time. For the time being I assumed it to be on, and therefore java.lang.String becomes str in my first version of the stub generator. Perhaps it would be more correct to treat it as Union[java.lang.String, str] (= either of these).
Still, a static type checker will not be able to detect if the conversion flag is going to be on or off at run-time.

[Thrameos]

The default for JPype 0.8 on will be conversion is false. The on mode is a bit of a misfeature as there are a few functions in Java where chaining a string is needed. (It is non obvious if you call the Java string constructor and get back a Python string). Though we may be able to splice some aliases into the java.lang.String so that it works a bit closer to Python strings. I remember there was a conflict on a method contract a while back. But I should probably refresh my memory on that topic.

We should definitely ship the stub generator as part of JPype so that someone can run it as needed in the field.

[Thrameos]

I reviewed the contract for string and Java string. There is one strong conflict, two weak conflicts and 2 near conflicts.

The strong conflict is Java and Python format. They are similar in design but one is a static method and the other operates on a string. Split is very similar but Java uses regexp rather than simple text on the split, Replace is almost the same but Python adds an extra parameter. endswith and endsWith are near miss. startswith and startsWith are near miss.

I could submit a PR with Java String completing almost the same contract as Python. The return type will always be Java strings for consistency. Give then we are switching to no longer convert strings, this would in principle save a few people, but as overloading strings completely is not possible (though maybe I should review this as this was last reviewed in Python 2 age. The other thing that will not pass is of course isinstance. Hard to dodge that one without deriving the type.

@marscher would this be worth the effort to put in as a PR?

[michi42]

Thanks a lot for checking. I agree that conceptually this feature is not very nice, although in many common use cases it saves a lot of explicit type conversion clutter. Checking how this is done in PySide (Qt), they do the same for QString ...
Actually I think there are two use cases for the automatic conversion - from str to java.lang.String (e.g. arguments of Java methods) and the other way around (e.g. return values of Java methods).

For the return types, yes, we may get away with making java.lang.String "look like" python str, and I agree it could be a nice feature, although as you noticed it will not cover all use cases.

On the other hand, what is the plan for str to java.lang.String conversion? This one seems "harmless" to me. With convertStrings=False, will you have to pass explicitly constructed java.lang.String objects to Java methods which take strings as arguments?

Anyway, I'm going to make that the stub generator honors the flag at generation time, so it is up to the user to decide. "Official" stubs would be generated assuming convertStrings=False, but for those using the feature, they can build their own stub tree with convertStrings=True. In the end it's a simple type replacement at the level of the stubs.

Edit: is there any foreseen way to detect from Python if JPype was started with convertStrings=True? Currently I'm using this, but it is somewhat artificial...

def convert_strings() -> bool:
    if convert_strings.jpype_flag is None:
        from java.lang import String  # noqa
        convert_strings.jpype_flag = isinstance(String().trim(), str)
    return convert_strings.jpype_flag


convert_strings.jpype_flag = None

[Thrameos]

Convert strings only affects the return path. We always implicitly convert string as arguments.

The argument path for stubs likely needs some extra hooks. We have quite a few automatic forward conversions like string, path, date, and list. I dont think there is currently a way to get a list of implicit conversions. Some of these are hard types, and others are duck types. What level of detail do you need for stubs? Do you need a method to extract the conversion rule by Java class?

[michi42]

Yes, then indeed I would need a way to extract a list of the types accepted by the implicit conversion for a particular java class in a method argument.
For the accepted hard types, it's straightforward, I would just need them as python type objects or strings.
For the duck types, we could define Protocol types - or if there is one, use one of the pre-defined protocols (e.g. Iterable, Mapping, ...)

Another question that goes in the same direction: is there a hook to get the "mangling" jpype does to identifiers when they match a python keywords (adding '_')?

[Thrameos]

The implicit rules currently recognize three types of conversions. The first is exact in which a conversion is only applied if it exactly matches, but this one is only there to trigger fast logic. The others are in the form of a list of types that will be taken or a list of attributes that will be probed. I suppose the best way to do this is to get you an API call that returns those two lists as a tuple so that we can construct Protocol types for each.

Name mangling is handled by the pysafe function in jpype._pykeywords. It is applied by classes to methods and fields, to accessors in beans, and to packages and imports.

[Thrameos]

Okay looking it over there is one special exception in the type system. Python strings are Sequences but JPype flatly refuses to recognize them. There is almost no case in which the user wants to pass a string to a list object and make it break it into chars.

[michi42]

Ok, I can't see how this special exception could be easily represented in the type annotation system, as if we e.g. allow Iterable, it will include str. But I would say, we can leave that aside for a first version.

[Thrameos]

"Never let perfect be the enemy of good."

There will also be an edge case or two. But nothing here gives me any pause. I will work on the probe API. Hopefully completed by the end of the weekend.

[michi42]

Perfect, thanks a lot, that would allow me to get rid of this hardcoded mess: https://gist.github.com/michi42/2110a8615a0e917a13ec6748c6168735#file-stubgenj-py-L212 ;-)

For objects returned by java to python, are there any implicit conversions (apart from String -> str if convertStrings = True)?
From what I see, boxed types are returned as such.
Primitives become JShort, JInt, ... - but since these implement the python int, it is probably safe to treat them as int (or bool, float, ... respectively)
Arrays become something that implements the List contract?

[Thrameos]

There are a few implicit conversions

• null goes to type None. So anything returning an object type will get None as well as the object type. Not sure stubs will care about this.
• Void type will eventually go to JVoid which will be magical type that refuses to do anything. This will prevent someone from accidentally using a void in an if statement.
• Java boolean goes to Python bool type (True, False)
• Java string goes to Python str when convertString is True
• Primitives each return the JPrimitive wrappers which all derive from the corresponding type.
• Boxes types act like primitives with the minor exception that they can be null objects. They do have some special methods so you may need to leave them as is.
• Java char goes to a str with length 1. This is a weird special case because I can't make str and int types play nice.
• Most containers will implement the corresponding contract and satisfy the collections.abc. This is still a work in progress. Best to leave them as the Java type then make the stub show it implements the contract.
• The other special case is the JObject cast which forces the type to match. Unfortunately it takes string arguments so we can really see the contract. The new casting operator fixes this problem. (Please comment on that thread if you have an operator preference).