[SPARK-34023][BUILD] Updated to Kryo 5.0.3

[nicknezis]

What changes were proposed in this pull request?

Updating Kryo to a newer version. Release information can be found here: https://github.com/EsotericSoftware/kryo/releases

Why are the changes needed?

Kryo 4.0.2 was released in March 2018. This 5.X release has many new features and performance improvements.

Does this PR introduce any user-facing change?

No

How was this patch tested?

mvn clean test

[dongjoon-hyun]

ok to test

[dongjoon-hyun]

Thank you for making a PR, @nicknezis .

[dongjoon-hyun]

I created SPARK-34023 for this.

[dongjoon-hyun]

cc @srowen

[LuciferYang]

@nicknezis Are there any exciting new features in Kryo 5.x?

[SparkQA]

Test build #133724 has finished for PR 31059 at commit 206bd63.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I don't think this actually updates the kryo version used? the pom.xml files were not changed.
This could be fine, but we'll have to see if it breaks anything once Spark actually uses it.

The migration guide is at https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
For example requiring registration by default may be a breaking change that we have to explicitly disable.

[nicknezis]

I don't think this actually updates the kryo version used? the pom.xml files were not changed.
This could be fine, but we'll have to see if it breaks anything once Spark actually uses it.

The migration guide is at https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
For example requiring registration by default may be a breaking change that we have to explicitly disable.

Yes I was a little bit unsure about this. The pom.xml files didn't seem to reference a version. And the previous PR #22179 that updated the version to 4.0.2 seemed to only touch these files. If you can let me know where the version information is actually stored, I can update that and redo any testing and follow-up code edits.

[srowen]

Yeah I'm pretty sure because it only comes in transitively via chill. Updating it might be a good thing, but, might break chill. Note that chill uses a shaded version of kryo. You can try to override the kryo-shaded dependency in the pom.xml file, but maybe better to see if there is a newer chill release that already uses it?

[nicknezis]

Ahh OK perfect. And is Chill the only thing that's bringing it in as a transitive dependency? I'll do more research down that path and look into upgrading it. I recently did the upgrade for Apache Heron and it wasn't that big of a change. If it isn't a simple chill version bump, should we close this pull request and keep SPARK-34023 for me to provide updates?

[srowen]

It looks like it's just chill. Spark doesn't use it directly though depends on its behavior. 0.9.5 looks like the latest chill release though. It may Just Work if you bump the kryo version, and could be worth a shot. If so that could be fine; only downside is now we manually manage the kryo version too, but, maybe that's OK. Yes keep the JIRA open / update the title/description.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38345/

[nicknezis]

Oh, I think I'm going to try to update Chill itself to support the newer Kryo. Then we can just bump the version of Chill in Spark. I don't think we should add the specific version of Kryo in Spark if it's not needed. I assume those dep files I manually updated are dynamically generated by some process? Is this something I would need to do as part of the process in updating Chill? This would explain why I was mistaken when referencing the older pull request that mentioned updating Kryo to 4.0.2.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38345/

[srowen]

Yes they're generated by the build process to detect unexpected changes in dependencies. There is a script in dev/ that can update it. But the file itself doesn't do anything beyond that check.

[SparkQA]

Test build #133757 has finished for PR 31059 at commit 0d96273.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

Thanks for working on this, @nicknezis !

Kryo 4.0.2 was released in March 2018. This 5.X release has many new features and performance improvements.

I'd like to see if this change can actually improves performance. If possible, could you run the kryo benchmark?

https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/serializer/KryoBenchmark.scala

[nicknezis]

Thanks for working on this, @nicknezis !

Kryo 4.0.2 was released in March 2018. This 5.X release has many new features and performance improvements.

I'd like to see if this change can actually improves performance. If possible, could you run the kryo benchmark?

https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/serializer/KryoBenchmark.scala

Until I get the opportunity to run the Spark benchmark, this Kryo benchmark might give some insight into what the changes might look like.

[nicknezis]

@srowen Sharing a link to the Google Groups discussion about what approach to take to update to Kyro 5 in Chill + Spark. I would really appreciate your input.

[srowen]

If updating chill means updating to kryo 5 and that means changing some imports in Spark, that's probably OK. Yes, kryo ends up in the user app namespace and that's a change that could be visible, but I don't think we expect users to rely on kryo directly. Still that might be a point up for discussion.

[nicknezis]

I'm sorry I lost steam on this. I plan to revisit, but got stuck in a morass of updating Chill, Storm and other dependencies. I think I have the Chill dependency updated. Working through the Spark changes now.