Scala 3.5.x stopped reporting generated classes via the Zinc incremental callback by default

[vasilmkd]

Compiler version

Scala 3.5.0-RC1 and above

I'm running Scala 3.5.0-RCX versions in the IntelliJ IDEA incremental compilation tests and I've noticed a change in behaviour. In IDEA, we have two modes of compilation, one using the Zinc incremental compiler, and one using the IDEA bytecode based incremental compiler.

Compiling using Zinc is not affected, because Zinc requires full analysis to run (xsbti.AnalysisCallback#enabled returns true).

When the alternative IDEA incremental compiler is used, we invoke scalac via sbt.internal.inc.AnalyzingCompiler. This is where I observe the change in behaviour.

All Scala versions prior to Scala 3.5.0 used to execute the xsbti.AnalysisCallback#generatedNonLocalClass and accompanying methods to report the production of .class files.

Judging by the previous behaviour, and the Scala 2 compiler source code, this method should be run regardless of the output of xsbti.AnalysisCallback#enabled. This can be seen here.

Somewhere around the changes made to the Scala 3 compiler when support for usePipelining was developed, this changed for Scala 3.5.0.

generatedNonLocalClass is now called only if xsbti.AnalysisCallback#enabled is true.

I will update this ticket as I discover more about the problem.

Tagging @bishabosha as someone who should have some context on the problem.

Thanks in advance.

[vasilmkd]

It's possible that this might be the commit.

Look for changes to ExtractAPI.scala.

[sjrd]

@bishabosha

[bishabosha]

so the breakdown of changes: #18137 (released in 3.4.0) introduced the change that ExtractAPI phase only runs if the callback is enabled, previously it only checked if the callback wasn't null

then commit a975398 (part of pipelining, so released in 3.5.0-RC1) moved notification of non-local classes from the backend phase, to the ExtractAPI phase - which due to the change above, means non-local classes were notified only when the callback is enabled.

A very simple fix would be to revert to running the ExtractAPI phase whenever the callback is merely present.

Otherwise we could restore reporting in backend when incremental phases are not enabled?

Or run a simplified collector in ExtractAPI to only collect classes for when we don't need incremental compile

[bishabosha]

@vasilmkd it would be good to know the behavior you observe with Scala 2, because it also only calls generatedNonLocalClass in the injected API phase, which as you showed, is only enabled when the callback is enabled

[vasilmkd]

When calling scalac through sbt.internal.inc.AnalyzingCompiler, we always pass a callback with enabled = false, regardless of the Scala version.
https://github.com/JetBrains/intellij-scala/blob/8e8495a31a79add1fb638060b81149b2d7db42ba/scala/compile-server/src/org/jetbrains/jps/incremental/scala/local/IdeaIncrementalCompiler.scala#L122

I have a test which runs some incremental compilation for many Scala versions. It only started failing when I added 3.5.0-RC1 to the mix.

https://github.com/JetBrains/intellij-scala/blob/8e8495a31a79add1fb638060b81149b2d7db42ba/scala/compiler-integration/test/org/jetbrains/plugins/scala/compiler/IncrementalCompilationTestBase.scala#L19-L33

Here's the test for reference.

@vasilmkd it would be good to know the behavior you observe with Scala 2, because it also only calls generatedNonLocalClass in the injected API phase, which as you showed, is only enabled when the callback is enabled

I'm now looking through Scala 2 code to verify

[vasilmkd]

Indeed, Scala 2.13 also doesn't run generatedNonLocalClass when enabled = false, I've verified that by manually debugging.

Let me then explain what the regression is in terms of the test expectations.

[vasilmkd]

The tests as can be seen here: https://github.com/JetBrains/intellij-scala/blob/8e8495a31a79add1fb638060b81149b2d7db42ba/scala/compiler-integration/test/org/jetbrains/plugins/scala/compiler/IncrementalCompilationTestBase.scala#L19-L33

org.jetbrains.plugins.scala.compiler.IncrementalCompilationTestBase#testRecompileOnlyAffectedFiles

A test project is set up with the following three source files.

// src/Fist.scala <- typo, but can be ignored
class First {
  def x = 1
}

// src/Second.scala
class Second extends First {
  println(x)
}

// src/Third.scala
class Third

The project is compiled as is, all .class files are produced.

The first source file is modified to be:

class First { def x = 1.0 }

The project is incrementally compiled. It is then asserted that First and Second (Second extends First) are recompiled, and Third is not recompiled.

The failure in this case is that Second.class and Second.tasty are not regenerated. It seems like Second.scala was not recompiled.

[vasilmkd]

org.jetbrains.plugins.scala.compiler.IncrementalCompilationTestBase#testDeleteOldTargetFiles

Again, a test project is created with the following sources:

src/First.scala
class First1
class First2

src/Second.scala
class Second

The project is compiled.
The source file src/First.scala is modified to (class First2 is removed):

class First1

The source file src/Second.scala is fully removed.
The project is incrementally compiled.

It is asserted that only First1.class and First1.tasty remain in the output directory, and that First2.class, First2.tasty, Second.class, Second.tasty have been removed.

With Scala 3.5.0-RCX, First2.class, First2.tasty, Second.class, Second.tasty all remain on disk. With all previous Scala versions, they are removed.