Compiler crashes when combining lazy val and macros

[WojciechMazur]

Similar to #20323 but this one is a regression introduced in 3.4.2
Based on OpenCB failure in shopstic/caliban - logs

Compiler version

Last good release: 3.4.2-RC1-bin-20240222-98efdab-NIGHTLY
First bad release: 3.4.2-RC1-bin-20240226-e0cb1e7-NIGHTLY

Bisect points to 5a884bc but compiler build deadlocked in a few revisions that were skipped, though it might not be accurate
Can be any of changes between 98efdab..e0cb1e7

Minimized code

// schema.scala
trait Schema[-R, T]
object Schema extends GenericSchema[Any]

trait GenericSchema[R] extends SchemaDerivation[R]
trait SchemaDerivation[A]:
  lazy val sampleDate: String = ??? // lazy val requried to reproduce

  inline def derived[R, A]: Schema[R, A] =
    lazy val annotations = Macros.annotations[A]
    new Schema[R, A] {}

  inline given gen[R, A]: Schema[R, A] = derived[R, A]

// macros.scala
import scala.quoted.*

object Macros:
  inline def annotations[T]: List[Any] = ${ annotationsImpl[T] }
  def annotationsImpl[T: Type](using qctx: Quotes): Expr[List[Any]] = Expr.ofList(Nil)

// usage.scala
sealed trait InputValue

class Usage:
  val inputValueSchema: Schema[Any, InputValue] = Schema.gen

Output (click arrow to expand)

  unhandled exception while running MegaPhase{elimErasedValueType, pureStats, vcElideAllocations, etaReduce, arrayApply, elimPolyFunction, tailrec, completeJavaEnums, mixin, lazyVals, memoize, nonLocalReturns, capturedVars} on /Users/wmazur/projects/sandbox/src/main/scala/usage.scala

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/lampepfl/dotty/issues/new/choose
  For non-enriched exceptions, compile with -Yno-enrich-error-messages.

     while compiling: /Users/wmazur/projects/sandbox/src/main/scala/usage.scala
        during phase: MegaPhase{elimErasedValueType, pureStats, vcElideAllocations, etaReduce, arrayApply, elimPolyFunction, tailrec, completeJavaEnums, mixin, lazyVals, memoize, nonLocalReturns, capturedVars}
                mode: Mode(ImplicitsEnabled)
     library version: version 2.13.12
    compiler version: version 3.4.2-RC1-bin-SNAPSHOT-git-5a884bc
            settings: -classpath /Users/wmazur/.ivy2/local/org.scala-lang/scala3-library_3/3.4.2-RC1-bin-SNAPSHOT/jars/scala3-library_3.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar -d /Users/wmazur/projects/sandbox/src/main/scala/.scala-build/scala_0de691b8c1/classes/main -java-output-version 17 -sourceroot /Users/wmazur/projects/sandbox/src/main/scala


  Exception while compiling /Users/wmazur/projects/sandbox/src/main/scala/macros.scala, /Users/wmazur/projects/sandbox/src/main/scala/test.scala, /Users/wmazur/projects/sandbox/src/main/scala/usage.scala

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/lampepfl/dotty/issues/new/choose
  For non-enriched exceptions, compile with -Yno-enrich-error-messages.

     while compiling: <no file>
        during phase: parser
                mode: Mode()
     library version: version 2.13.12
    compiler version: version 3.4.2-RC1-bin-SNAPSHOT-git-5a884bc
            settings: -classpath /Users/wmazur/.ivy2/local/org.scala-lang/scala3-library_3/3.4.2-RC1-bin-SNAPSHOT/jars/scala3-library_3.jar:/Users/wmazur/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar -d /Users/wmazur/projects/sandbox/src/main/scala/.scala-build/scala_0de691b8c1/classes/main -java-output-version 17 -sourceroot /Users/wmazur/projects/sandbox/src/main/scala

Exception in thread "main" dotty.tools.dotc.core.Denotations$StaleSymbolException: stale symbol; val OFFSET$_m_0#27281 in module class Schema$, defined in Period(2.78-86), is referred to in run Period(3.78)
        at dotty.tools.dotc.core.Denotations$SingleDenotation.staleSymbolError(Denotations.scala:960)
        at dotty.tools.dotc.core.Denotations$SingleDenotation.bringForward(Denotations.scala:758)
        at dotty.tools.dotc.core.Denotations$SingleDenotation.toNewRun$1(Denotations.scala:805)
        at dotty.tools.dotc.core.Denotations$SingleDenotation.current(Denotations.scala:876)
        at dotty.tools.dotc.core.Symbols$Symbol.recomputeDenot(Symbols.scala:124)
        at dotty.tools.dotc.core.Symbols$Symbol.computeDenot(Symbols.scala:118)
        at dotty.tools.dotc.core.Symbols$Symbol.denot(Symbols.scala:109)
        at dotty.tools.dotc.core.Symbols$.toDenot(Symbols.scala:542)
        at dotty.tools.dotc.transform.Mixin.traitInits$1$$anonfun$1(Mixin.scala:264)
        at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:479)
        at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
        at scala.collection.immutable.List.prependedAll(List.scala:152)
        at scala.collection.immutable.List$.from(List.scala:684)
        at scala.collection.immutable.List$.from(List.scala:681)
        at scala.collection.IterableOps$WithFilter.map(Iterable.scala:898)
        at dotty.tools.dotc.transform.Mixin.traitInits$1(Mixin.scala:285)
        at dotty.tools.dotc.transform.Mixin.$anonfun$6(Mixin.scala:315)
        at scala.collection.immutable.List.flatMap(List.scala:293)
        at dotty.tools.dotc.transform.Mixin.transformTemplate(Mixin.scala:318)
        at dotty.tools.dotc.transform.Mixin.transformTemplate(Mixin.scala:184)
        at dotty.tools.dotc.transform.MegaPhase.goTemplate(MegaPhase.scala:1067)
        at dotty.tools.dotc.transform.MegaPhase.goTemplate(MegaPhase.scala:1068)
        at dotty.tools.dotc.transform.MegaPhase.goTemplate(MegaPhase.scala:1068)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:373)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:268)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:448)
        at dotty.tools.dotc.transform.MegaPhase.loop$2(MegaPhase.scala:467)
        at dotty.tools.dotc.transform.MegaPhase.transformBlock(MegaPhase.scala:472)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:311)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.loop$2$$anonfun$1(MegaPhase.scala:470)
        at dotty.tools.dotc.transform.MegaPhase.loop$2(MegaPhase.scala:472)
        at dotty.tools.dotc.transform.MegaPhase.transformBlock(MegaPhase.scala:472)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:311)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:333)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:333)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.loop$2$$anonfun$1(MegaPhase.scala:470)
        at dotty.tools.dotc.transform.MegaPhase.loop$2(MegaPhase.scala:472)
        at dotty.tools.dotc.transform.MegaPhase.transformBlock(MegaPhase.scala:472)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:311)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.mapDefDef$1(MegaPhase.scala:261)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:264)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:448)
        at dotty.tools.dotc.transform.MegaPhase.loop$1(MegaPhase.scala:461)
        at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:461)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:372)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:268)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:448)
        at dotty.tools.dotc.transform.MegaPhase.loop$1(MegaPhase.scala:461)
        at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:461)
        at dotty.tools.dotc.transform.MegaPhase.mapPackage$1(MegaPhase.scala:392)
        at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:395)
        at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:450)
        at dotty.tools.dotc.transform.MegaPhase.transformUnit(MegaPhase.scala:477)
        at dotty.tools.dotc.transform.MegaPhase.run(MegaPhase.scala:489)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:354)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:360)
        at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:315)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:337)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:350)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:360)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:360)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:267)
        at dotty.tools.dotc.Run.compileSuspendedUnits(Run.scala:371)
        at dotty.tools.dotc.Driver.finish(Driver.scala:57)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:38)
        at dotty.tools.dotc.Driver.process(Driver.scala:196)
        at dotty.tools.dotc.Driver.process(Driver.scala:164)
        at dotty.tools.dotc.Driver.process(Driver.scala:176)
        at dotty.tools.dotc.Driver.main(Driver.scala:206)
        at dotty.tools.dotc.Main.main(Main.scala)

[KacperFKorban]

@Gedochao Do you mind if I reassign this to myself? (@jchyb is away right now anyway)
Plus this seems unrelated to metaprogramming to me. It seems to me that the only thing the macro expansion changes is incrementing the runId for the usage file. Also, the affecting PR is this one: #19786, specifically the recomputeDenot change

Also', here is a smaller minimization:

// Macro.scala
import scala.quoted.*

trait Schema
object Schema:
  lazy val sampleDate: String = "" // lazy val requried to reproduce

  inline def derived: Schema =
    annotations
    new Schema {}

inline def annotations: Int = ${ annotationsImpl }
def annotationsImpl(using Quotes): Expr[Int] = Expr(1)

// Test.scala
//> using scala 3.nightly

val inputValueSchema = Schema.derived

Output:

-- [E197] Potential Issue Warning: ~/bugs/i21271/Macro.scala:10:4 ------
10 |    new Schema {}
   |    ^
   |   New anonymous class definition will be duplicated at each inline site
   |
   | longer explanation available when compiling with `-explain`
1 warning found

  unhandled exception while running MegaPhase{elimErasedValueType, pureStats, vcElideAllocations, etaReduce, arrayApply, elimPolyFunction, tailrec, completeJavaEnums, mixin, lazyVals, memoize, nonLocalReturns, capturedVars} on ~/bugs/i21271/Test.scala

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.

     while compiling: ~/bugs/i21271/Test.scala
        during phase: MegaPhase{elimErasedValueType, pureStats, vcElideAllocations, etaReduce, arrayApply, elimPolyFunction, tailrec, completeJavaEnums, mixin, lazyVals, memoize, nonLocalReturns, capturedVars}
                mode: Mode(ImplicitsEnabled)
     library version: version 2.13.14
    compiler version: version 3.6.0-RC1-bin-20240724-f0b3a2b-NIGHTLY-git-f0b3a2b
            settings: -classpath ~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.6.0-RC1-bin-20240724-f0b3a2b-NIGHTLY/scala3-library_3-3.6.0-RC1-bin-20240724-f0b3a2b-NIGHTLY.jar:~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar -d ~/bugs/i21271/.scala-build/i21271_73afa7fde9/classes/main -java-output-version 17 -sourceroot ~/bugs/i21271


  Exception while compiling ~/bugs/i21271/Macro.scala, ~/bugs/i21271/Test.scala

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.

     while compiling: <no file>
        during phase: parser
                mode: Mode()
     library version: version 2.13.14
    compiler version: version 3.6.0-RC1-bin-20240724-f0b3a2b-NIGHTLY-git-f0b3a2b
            settings: -classpath ~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.6.0-RC1-bin-20240724-f0b3a2b-NIGHTLY/scala3-library_3-3.6.0-RC1-bin-20240724-f0b3a2b-NIGHTLY.jar:~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.14/scala-library-2.13.14.jar -d ~/bugs/i21271/.scala-build/i21271_73afa7fde9/classes/main -java-output-version 17 -sourceroot ~/bugs/i21271

Exception in thread "main" dotty.tools.dotc.core.Denotations$StaleSymbolException: stale symbol; val OFFSET$_m_0#36284 in module class Schema$, defined in Period(2.78-86), is referred to in run Period(3.78)
	at dotty.tools.dotc.core.Denotations$SingleDenotation.staleSymbolError(Denotations.scala:961)
	at dotty.tools.dotc.core.Denotations$SingleDenotation.bringForward(Denotations.scala:759)
	at dotty.tools.dotc.core.Denotations$SingleDenotation.toNewRun$1(Denotations.scala:806)
	at dotty.tools.dotc.core.Denotations$SingleDenotation.current(Denotations.scala:877)
	at dotty.tools.dotc.core.Symbols$Symbol.recomputeDenot(Symbols.scala:124)
	at dotty.tools.dotc.core.Symbols$Symbol.computeDenot(Symbols.scala:118)
	at dotty.tools.dotc.core.Symbols$Symbol.denot(Symbols.scala:109)
	at dotty.tools.dotc.core.Symbols$.toDenot(Symbols.scala:544)
	at dotty.tools.dotc.transform.Mixin.traitInits$1$$anonfun$1(Mixin.scala:266)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:479)
	at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
	at scala.collection.immutable.List.prependedAll(List.scala:152)
	at scala.collection.immutable.List$.from(List.scala:685)
	at scala.collection.immutable.List$.from(List.scala:682)
	at scala.collection.IterableOps$WithFilter.map(Iterable.scala:900)
	at dotty.tools.dotc.transform.Mixin.traitInits$1(Mixin.scala:265)
	at dotty.tools.dotc.transform.Mixin.$anonfun$6(Mixin.scala:325)
	at scala.collection.immutable.List.flatMap(List.scala:294)
	at dotty.tools.dotc.transform.Mixin.transformTemplate(Mixin.scala:320)
	at dotty.tools.dotc.transform.Mixin.transformTemplate(Mixin.scala:186)
	at dotty.tools.dotc.transform.MegaPhase.goTemplate(MegaPhase.scala:1071)
	at dotty.tools.dotc.transform.MegaPhase.goTemplate(MegaPhase.scala:1072)
	at dotty.tools.dotc.transform.MegaPhase.goTemplate(MegaPhase.scala:1072)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:377)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:272)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
	at dotty.tools.dotc.transform.MegaPhase.loop$2(MegaPhase.scala:471)
	at dotty.tools.dotc.transform.MegaPhase.transformBlock(MegaPhase.scala:476)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:315)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
	at dotty.tools.dotc.transform.MegaPhase.loop$2$$anonfun$1(MegaPhase.scala:474)
	at dotty.tools.dotc.transform.MegaPhase.loop$2(MegaPhase.scala:476)
	at dotty.tools.dotc.transform.MegaPhase.transformBlock(MegaPhase.scala:476)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:315)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:337)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
	at dotty.tools.dotc.transform.MegaPhase.mapDefDef$1(MegaPhase.scala:265)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:268)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
	at dotty.tools.dotc.transform.MegaPhase.loop$1(MegaPhase.scala:465)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:465)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:376)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:272)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:452)
	at dotty.tools.dotc.transform.MegaPhase.loop$1(MegaPhase.scala:465)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:465)
	at dotty.tools.dotc.transform.MegaPhase.mapPackage$1(MegaPhase.scala:396)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:399)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:454)
	at dotty.tools.dotc.transform.MegaPhase.transformUnit(MegaPhase.scala:481)
	at dotty.tools.dotc.transform.MegaPhase.run(MegaPhase.scala:493)
	at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:380)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.List.foreach(List.scala:334)
	at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:373)
	at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
	at dotty.tools.dotc.Run.runPhases$1(Run.scala:336)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:384)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:396)
	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:396)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:288)
	at dotty.tools.dotc.Run.compileSuspendedUnits(Run.scala:410)
	at dotty.tools.dotc.Driver.finish(Driver.scala:63)
	at dotty.tools.dotc.Driver.doCompile(Driver.scala:38)
	at dotty.tools.dotc.Driver.process(Driver.scala:201)
	at dotty.tools.dotc.Driver.process(Driver.scala:169)
	at dotty.tools.dotc.Driver.process(Driver.scala:181)
	at dotty.tools.dotc.Driver.main(Driver.scala:211)
	at dotty.tools.dotc.Main.main(Main.scala)
Compilation failed

[jchyb]

When a macro definition and call are located in the same compilation run, the compiler suspends the macro call compilation unit, finishes compiling other stuff and comes back to the macro later. Some parts of the context and symbol denotations can be shared between those runs which can cause those stale symbol issues. I believe there is might be a pass refreshing denotations somewhere in typer for exactly that occasion, but I still don't quite understand everything there. With that said please feel free to try taking this on if you want @KacperFKorban, especially since you have a lead. I find those stale symbol errors very difficult personally, so any help is appreciated