kotlin_rules and Ijars: Update the Ijar tooling to do the right thing.

[hsyed]

I have been working on a new set of kotlin rules and they are currently being migrated to bazelbuild/rules_kotlin.

One goal with the design of the Kotlin JVM ruleset is to stay as close to the native java rules and infrastructure as possible. The automatic Ijars generation is problematic as the java rules auto create ijars. Ijars generated for Kotlin work in a lot of cases, but inlined methods break (might be other cases).

Proposal: Adapt Ijar to do the right thing for Kotlin Jars.

I propose making a change to the behaviour of the ijar tool so that it does the right thing when it comes to Kotlin metadata. Adding this change to Bazel core simplifies the Kotlin rules and allows more reuse out of java common infrastructure. I propose the ijar tool be updated to:

1. Initially to detect a Kotlin jar and ignore it for processing -- perhaps just symlink the ijar to the full jar ? This should be straight forward a Kotlin jar has some markers that can be identified without processing bytecode.
2. Adapt the tool to generate valid Kotlin Ijars. Afaik this means retaining certain bytecode elements and marker files.

Motivating example 1: dep management tooling / java_import.

I have just added a kotlin_import rule to:

1. Get around the automatic ijar generation by java_import, and
2. Ensure the Jar has a KotlinInfo provider so that full_compile_jars are always selected when compiling Kotlin.

Any fancy java dependency management tooling that we develop will likely work via orchestrations of java_import, java_library and java_import_external it's a shame to have to add a layer for Kotlin when it's barely needed.

[meteorcloudy]

@iirina Can you take a look at this?

[iirina]

Can you explain why the automatic ijar generation in the java rules is problematic? Is it because you are using java_import to import kotlin jars, in which case ijar will fail (sometimes)? How does it impact java_library/java_import_external?

[hsyed]

I misrepresented the sometimes part. Most Kotlin libraries that a JVM project would use as an external dependency will have DSLs that make heavy use of inline. So some intermediate kotlin_ rule has to be used to:

• Add a KotlinInfo -- to enable rules in the workspace to ensure full_compile_jars are used when using the target as a dependency.
• Force selection of full_compile_jars by re-interpreting the JavaInfo from deps imported as plain Java Jars --e.g., using the //jar component of maven_jar.

If the Kotlin ABI elements were retained by the Ijar tool transparently then special handling would not be needed so Kotlin libraries could be imported by java_import, maven_jar, java_library.

Current dependency management strategies (pubref/rules_maven, bazel-dep, migration-tooling, this one) go beyond this. They generate BUILD file entries which range from from simply listing runtime_deps to modelling the transitive deps in fine grained detail (exports etc).

Current and future versions of these tools could work for Kotlin transparently as long as the ijar tool doesn't strip off the Kotlin ABI elements.

This benefits of this feature go beyond just the motivational example I gave above. Some other reasons for why updating the ijar tool:

• The intellij plugin model has to have conditional checks for the intermediate rules --e.g., kotlin_import and kotlin_stdlib.

[cushon]

Adapt the tool to generate valid Kotlin Ijars. Afaik this means retaining certain bytecode elements and marker files.

Is there a reference for specifically how these ABI elements are represented in bytecode?

[hsyed]

Just had a closer look.

For marker files there is a META-INF/<module name>.kotlin_module. Might be more depending on the compilation modes / module types.

After some experimentation inlining metadata seems to be embedded in the Metadata annotation. Every class has a Metadata annotation.

Using the bytecode tool included in the Intellij Kotlin plugin to inspect the following test file:

package test

fun monkey(x: () -> Unit) {
    x()
}

inline fun donkey(y: () -> Unit): Int {
    y()
    return 1
}

fun lonkey(z: () -> Unit): Int {
    z()
}

d2 entry in metadata -- notice the inlined function includes the param type.

d2={"donkey", "", "y", "Lkotlin/Function0;", "", "lonkey", "z", "monkey", "x", "production sources for module .workspace"})

full output

// ================test/TestKt.class =================
// class version 50.0 (50)
// access flags 0x31
public final class test/TestKt {


  // access flags 0x19
  // signature (Lkotlin/jvm/functions/Function0<Lkotlin/Unit;>;)V
  // declaration: void monkey(kotlin.jvm.functions.Function0<kotlin.Unit>)
  public final static monkey(Lkotlin/jvm/functions/Function0;)V
    @Lorg/jetbrains/annotations/NotNull;() // invisible, parameter 0
   L0
    ALOAD 0
    LDC "x"
    INVOKESTATIC kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull (Ljava/lang/Object;Ljava/lang/String;)V
   L1
    LINENUMBER 4 L1
    ALOAD 0
    INVOKEINTERFACE kotlin/jvm/functions/Function0.invoke ()Ljava/lang/Object;
    POP
   L2
    LINENUMBER 5 L2
    RETURN
   L3
    LOCALVARIABLE x Lkotlin/jvm/functions/Function0; L0 L3 0
    MAXSTACK = 2
    MAXLOCALS = 1

  // access flags 0x19
  // signature (Lkotlin/jvm/functions/Function0<Lkotlin/Unit;>;)I
  // declaration: int donkey(kotlin.jvm.functions.Function0<kotlin.Unit>)
  public final static donkey(Lkotlin/jvm/functions/Function0;)I
    @Lorg/jetbrains/annotations/NotNull;() // invisible, parameter 0
   L0
    ALOAD 0
    LDC "y"
    INVOKESTATIC kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull (Ljava/lang/Object;Ljava/lang/String;)V
   L1
    LINENUMBER 8 L1
    ALOAD 0
    INVOKEINTERFACE kotlin/jvm/functions/Function0.invoke ()Ljava/lang/Object;
    POP
   L2
    LINENUMBER 9 L2
    ICONST_1
    IRETURN
   L3
    LOCALVARIABLE y Lkotlin/jvm/functions/Function0; L0 L3 0
    LOCALVARIABLE $i$f$donkey I L0 L3 1
    MAXSTACK = 2
    MAXLOCALS = 2

  // access flags 0x19
  // signature (Lkotlin/jvm/functions/Function0<Lkotlin/Unit;>;)I
  // declaration: int lonkey(kotlin.jvm.functions.Function0<kotlin.Unit>)
  public final static lonkey(Lkotlin/jvm/functions/Function0;)I
    @Lorg/jetbrains/annotations/NotNull;() // invisible, parameter 0
   L0
    ALOAD 0
    LDC "z"
    INVOKESTATIC kotlin/jvm/internal/Intrinsics.checkParameterIsNotNull (Ljava/lang/Object;Ljava/lang/String;)V
   L1
    LINENUMBER 13 L1
    ALOAD 0
    INVOKEINTERFACE kotlin/jvm/functions/Function0.invoke ()Ljava/lang/Object;
    POP
   L2
    LINENUMBER 14 L2
    ICONST_0
    IRETURN
   L3
    LOCALVARIABLE z Lkotlin/jvm/functions/Function0; L0 L3 0
    MAXSTACK = 2
    MAXLOCALS = 1

  @Lkotlin/Metadata;(mv={1, 1, 9}, bv={1, 0, 2}, k=2, d1={"\u0000\u0014\n\u0000\n\u0002\u0010\u0008\n\u0000\n\u0002\u0018\u0002\n\u0002\u0010\u0002\n\u0002\u0008\u0005\u001a\u0017\u0010\u0000\u001a\u00020\u00012\u000c\u0010\u0002\u001a\u0008\u0012\u0004\u0012\u00020\u00040\u0003H\u0086\u0008\u001a\u0014\u0010\u0005\u001a\u00020\u00012\u000c\u0010\u0006\u001a\u0008\u0012\u0004\u0012\u00020\u00040\u0003\u001a\u0014\u0010\u0007\u001a\u00020\u00042\u000c\u0010\u0008\u001a\u0008\u0012\u0004\u0012\u00020\u00040\u0003\u00a8\u0006\u0009"}, d2={"donkey", "", "y", "Lkotlin/Function0;", "", "lonkey", "z", "monkey", "x", "production sources for module .workspace"})
  // compiled from: test.kt
}


// ================META-INF/production sources for module .workspace.kotlin_module =================
���������������	
�
�test��TestKt

I think I read somewhere that the bytecode contained within the stub of an inlined function is used as well.

references:

understand every line of your codebase

kotlin-bytecode-generation-and-runtime-performance

[cushon]

ijar will remove non-class files including META-INF/<module name>.kotlin_module, but that would be easy to change.

It does not remove annotations, so @Metadata should already be preserved.

For inline methods, I assume the implementation needs to be visible across compilation boundaries? It might need a way to identify inline methods and to include their implementations in the interface jar.

[hsyed]

Yes I think thats right, the compiler seems to generate bytecode mimicking a regular JDK8 higher order function, the lambda invocation instructions must be used as the placeholders, nice design.

It seems like a simple enough change, visit classes with Metadata annotations, discover public and protected inline methods and preclude them from bytecode dropping.

I could gather a list of the file patterns relevant to the various module types -- as globs ?.

---

Since we are visiting the jars and the metadata I wonder if it would be possible to either:

1. Attach providers marking the Kotlin Jar type, along with some basic fields such as module name.
2. Or just attach a basic provider marking the Kotlin Jar type and write out a bazel proto file containing relevant metadata.

There are going to be quite a few rule and rule type variants. If we had option 1 it would really simplify the design as we can constraint the attributes.

[lberki]

In my uninformed opinion, I think it's fine to special-case Kotlin stuff in ijar a little bit; the above "don't drop bytecode from methods that are marked as inline" counts as such in my book.

I don't understand the latter half of @hsyed 's comment, though -- how is this new proposed provider relevant to changes to ijar?

[hsyed]

@iberki I've removed the second part from my previous comment. It's not relevant to this issue at all.

[davidstanke]

Great -- sounds like we're well aligned on approach. @hsyed do you have the bandwidth to implement this? @cushon any concerns about the approach described?

[lberki]

@hsyed -- excellent, I was afraid for a second that my understanding is that hazy :)