Scalac classpath cache should invalidate modified jars

[smarter]

Since Scala 2.12, classpath entries are cached unless -YdisableFlatCpCaching is passed to scalac, this can have very surprising results like breaking this sbt test that uses unmanagedJars: https://github.com/sbt/sbt/tree/0.13/sbt/src/sbt-test/source-dependencies/binary (you can copy this folder somewhere, set scalacOptions to 2.12.2 and verify that launching sbt, doing use/compile, removing dep/A.scala, and doing use/compile again compiles even though the dependency does not exist and the jar has been updated, adding scalacOptions += "-YdisableFlatCpCaching" fixes that.

I think the scalac cache in https://github.com/scala/scala/blob/2.12.x/src/compiler/scala/tools/nsc/classpath/ZipAndJarFileLookupFactory.scala should be more intelligent and invalidate modified jars.

[eed3si9n]

+1

Check out sbt/zinc#303 / sbt/zinc#282.
sbt has a setting called exportJars that exposes subproject dependencies as JAR file. When combined with Scala 2.12.2 it no longer invalidates the user source file, so that's a surprising behavior.

ZipAndJarFileLookupFactory.scala should be more intelligent and invalidate modified jars.

I'd echo @smarter's comment here. Overall the idea of caching makes sense, but it probably needs to check the timestamp or SHA-1 or something to not break existing builds.

[jvican]

A huge thumbs up to fixing this issue. I want to implement this sbt/zinc#305 in Zinc at some point, and I prefer not to ask users of it to add scalacOptions += "-YdisableFlatCpCaching".

On an independent topic, is there any particular reason why this caching of jars is performed by Scalac? How big was the performance improvement?

[smarter]

On an independent topic, is there any particular reason why this caching of jars is performed by Scalac? How big was the performance improvement?

scala/scala#4060 gives some background, the main usecase is IDEs where each subproject will get its own compiler instance:

I built Scala IDE using my version of Scala and tested it with a very large codebase with about 100 projects. I turned off the closing of unused presentation compilers which I added lately. And after that I did certain custom operation which does a deep refresh of all projects (close projects, regenerate the workspace, open projects).

I've also observed the dotty test suite running faster after merging the 2.12 classpath implementation in dotty: scala/scala3#2191

[jvican]

Thanks for the clarification @smarter.

[smarter]

Ping @adriaanm @retronym , it'd be great if someone could have a look at this issue.

[adriaanm]

@lrytz, what do you think?

[retronym]

This cache does appear to overreach, I think can we should tame it for 2.12.3. A minimal change would be to flip the default value of the disableCaching setting to true, and set it to false in the presentation compiler (assuming the user hasn't explicitly set it).

A more nuanced change would be to use file timestamps/size/content-hash, or notifications from the nio.file.WatchService to invalidate it.

It also represents a memory leak in its current form.

[retronym]

WIP: https://github.com/retronym/scala/tree/ticket/10295