Unpacked file size is nearly at 50 MB

[phkorn]

Hi,
we are using cats-core on AWS Lambda.
After hitting the 250 MB hard limit, I extracted our fat jar and found out that cats is our biggest dependency by a stretch.

Is there any way to reduce this?
I know there is no tree-shaking as with other languages and some pointed me to using protector.

[joroKr21]

Can you provide more details? Which versions of Cats/Scala, which platform?
When I look at Maven Central, the jar is ~6MB: https://mvnrepository.com/artifact/org.typelevel/cats-core_2.13/2.13.0

[phkorn]

We are using cats-core:2.13.0

Getting file size

curl -O https://repo1.maven.org/maven2/org/typelevel/cats-core_3/2.13.0/cats-core_3-2.13.0.jar

du -h cats-core_3-2.13.0.jar
# 6.8M    cats-core_3-2.13.0.jar

unzip cats-core_3-2.13.0.jar

du -sh cats
# 30M    cats

When extracting our fat jar MacOS finder tells me the following for the cats folder
42.041.296 bytes (50,6 MB on disk)

We are using sbt assembly for generating the fat jar.

[joroKr21]

A few things to note:

• You said you are using Scala 2.13 but you are showing the jar size for Scala 3
• The terminal is using MiB as a unit and Finder is using MB as a unit so they show different numbers

[joroKr21]

I think ApplyArityFunctions generates a lot of bridge methods

[satorg]

I guess there are lots of other arity-specifc traits in Cats that contribute to the inflated module size too, e.g.: Tuple[1|2|...|22]SemigroupalOps, ParallelArityFunctions, just to name a few.

A sad part here is that most of the higher-arity traits and methods are rarely used, but sit in the jar anyway.
Not sure what we can do about it though.

Perhaps, one of possible solutions could be generating two different cats-core artifacts, e.g. cats-core itself as a "full-sized" one, and cats-core-min as a minimalistic version that only contains arities up to some modest number, e.g. 5 or 6.

Another part that could be somehow optimized: there are some non-essential data types like NonEmptyList, NonEmptyVector, etc that might not be exactly required for everyone who depends on cats-core for its typeclasses only. Therefore, such data classes could be extracted into a separate module in order to reduce the size of the core one, I think.

[johnynek]

I don't know how common this issue is, but it seems like it might be a considerable amount of work to address well for everyone.

One thing I'd suggest is to look into using jar jar rules (see: https://eed3si9n.com/jarjar-abrams/ ) in your build to remove the arity functions.

In fact with a bit of tuning you can probably remove a huge amount from your jars before deploy.

Yes it's more work, but also you are in control of it.

If you don't use any reflection transitively in your app, you can possibly use the automatic removal of classes just by declaring the main class entry point.

[joroKr21]

I guess there are lots of other arity-specifc traits in Cats that contribute to the inflated module size too, e.g.: Tuple[1|2|...|22]SemigroupalOps, ParallelArityFunctions, just to name a few.

The problem are not just the arity functions by themselves. The issue is that they are generated in traits and those traits generate bridge methods in every subclass. They should have been generated as abstract classes.

[satorg]

The problem was addressed (to some extent) in #4751.