Compilation Performance

[tinder-aminghadersohi]

The latest version of the library is actually not usable due to super slow compile times.

gw :model-generation:app:kotlin:build
<=======------> 59% EXECUTING [16m 10s]
> :model-generation:app:kotlin:compileKotlin
^C%

This is on the latest macbook pro with max specs.
Is there any way to disable descriptor and json support?

[garyp]

JSON support is implemented 100% in the runtime library now, so it shouldn't impact compilation of generated code. Descriptors, which are in the generated code, are used to power both binary and JSON encoding/decoding so it wouldn't really make sense to disable them. If they're too slow to compile though then I'd definitely like to see how we can optimize them. Which Kotlin version is this with?

[garyp]

Can you also try this fix in CodeGenerator.kt from the kotlin-1.4 branch and see if it improves compile times for you: 7b9c2cf#diff-e06b3a8501dd08fb14559fe0145935ad ?

[tinder-aminghadersohi]

pbandk 0.9.0-rc.1
kotlin 1.4.10
kotlinx-serialization-json 1.0.0-RC2

I will try the fix you suggested

[tinder-aminghadersohi]

I see you are using the same deps in that suggested commit.

[garyp]

Yes. That's the branch to update pbandk to build with Kotlin 1.4. There are a couple Gradle metadata issues I'm still struggling to figure out with the Kotlin 1.4 update. Once I've got those resolved I'm planning to merge that branch into master (hopefully for the 0.10.0 release).

However, code generated by pbandk should be buildable with Kotlin 1.4 even if pbandk is still being built with Kotlin 1.3. So if there are any issues in the generated code (such as your current issue), then I'd like to try to get them fixed for 0.9.0.

[tinder-aminghadersohi]

The compile time increase from 2m to 16m+ is a deal breaker for my upgrade to pbandk 0.9.0. I will have to put it on hold until we have a solution.

[garyp]

Did the fix from that commit not help? You don't need to grab the entire commit. Just the changes to CodeGenerator.kt.

Do you have any insights into which aspect of your models might be out of the ordinary and responsible for the long compilation times? For example, large number of fields? Or large number of oneof cases? Or large number of messages in a single proto file? Etc. That would help me know which aspect of the code generator to focus on optimizing.

[tinder-aminghadersohi]

The commit uses the same versions that i tried last and the compile times were really bad. We have about 1000 protobuf messages one of which has about 500 sub messages, and another oneof with 150 sub messages. I can try to comment those out and see if thats the bottleneck

[tinder-aminghadersohi]

A oneof with 50 messages has a reasonable compile time of about 1-2 minutes. Once that hit 100 messages it killed intelij - i was running gradle in the integrate terminal

[garyp]

Thanks for narrowing down the cause @tinder-aminghadersohi. I'll try to reproduce it on my end so I can figure out where the regression is.

Do you know if this happens when you compile with Kotlin 1.3 too? I've noticed some performance regressions in the Kotlin 1.4 compiler with code that used to work fine under Kotlin 1.3.

The commit uses the same versions that i tried last and the compile times were really bad.

Sorry, I didn't quite understand your response. Did you patch pbandk's CodeGenerator.kt file as I suggested (and build a custom version of protoc-gen-kotlin) and you still saw the performance problem? Or did you only update the Kotlin versions but not patch CodeGenerator.kt?

[tinder-aminghadersohi]

Thanks for narrowing down the cause @tinder-aminghadersohi. I'll try to reproduce it on my end so I can figure out where the regression is.

Cool, will you just generated 100+ random proto messages with a dozen or more attribs?

Do you know if this happens when you compile with Kotlin 1.3 too? I've noticed some performance regressions in the Kotlin 1.4 compiler with code that used to work fine under Kotlin 1.3.

I dont think i tried 1.3 but i can later today.

The commit uses the same versions that i tried last and the compile times were really bad.

Sorry, I didn't quite understand your response. Did you patch pbandk's CodeGenerator.kt file as I suggested (and build a custom version of protoc-gen-kotlin) and you still saw the performance problem? Or did you only update the Kotlin versions but not patch CodeGenerator.kt?

I didn't try the CodeGenerator.kt patch but I can do that later today also.

[garyp]

@tinder-aminghadersohi Did you have a chance to try the CodeGenerator.kt patch? I was able to reproduce your problem under Kotlin 1.4.10 with a very large protobuf message definition. That same .proto file built fine under Kotlin 1.3, so it does seem like a Kotlin 1.4 regression. Applying the CodeGenerator.kt patch fixed the problem for me. I've put up that patch in a PR: #101. Can you please try out that PR to see if it fixes the problem you were seeing?

[garyp]

@tinder-aminghadersohi The fix has been merged into master. If you still see problems with the fix applied, please feel free to reopen this issue.