Too many open files when running `dotty.tools.dotc.IdempotencyTest.idempotency`.

[Duhemm]

On my machine, running the test dotty.tools.dotc.IdempotencyTests.idempotency, because there are too many files open.

> testOnly dotty.tools.dotc.IdempotencyTests -- --tests=idempotency

First bad commit: 92a25cb
ref #3395

Partial stack trace

[info] Test dotty.tools.dotc.IdempotencyTests.idempotency started
[=================================>     ] completed (1553/1776, 71s)
error while loading Short,
 /Library/Java/Extensions: Too many open files

error while loading Integer,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files

error while loading Throwable,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files
java.nio.file.FileSystemException: /Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/SB.tasty: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at dotty.tools.io.AbstractFile.fileOrSubdirectoryNamed(AbstractFile.scala:247)
        at dotty.tools.io.AbstractFile.fileNamed(AbstractFile.scala:258)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile(BytecodeWriters.scala:35)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile$(BytecodeWriters.scala:28)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFile(GenBCode.scala:59)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile(BCodeHelpers.scala:38)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile$(BCodeHelpers.scala:37)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFileForClassfile(GenBCode.scala:59)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.$anonfun$visit$2(GenBCode.scala:209)
        at scala.Option.foreach(Option.scala:257)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.visit(GenBCode.scala:204)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.run(GenBCode.scala:152)
        at dotty.tools.backend.jvm.GenBCodePipeline.buildAndSendToDisk(GenBCode.scala:367)
        at dotty.tools.backend.jvm.GenBCodePipeline.run(GenBCode.scala:333)
        at dotty.tools.backend.jvm.GenBCode.run(GenBCode.scala:54)
        at dotty.tools.dotc.core.Phases$Phase.$anonfun$runOn$1(Phases.scala:289)
        at scala.collection.immutable.List.map(List.scala:283)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:287)
        at dotty.tools.dotc.core.Phases$Phase.runOn$(Phases.scala:286)
        at dotty.tools.backend.jvm.GenBCode.runOn(GenBCode.scala:38)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$3(Run.scala:125)

error while loading Long,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.trackTime(Stats.scala:47)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2(Run.scala:123)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2$adapted(Run.scala:121)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:121)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$1(Run.scala:141)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:86)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:107)
        at dotty.tools.dotc.Run.compileSources(Run.scala:94)
        at dotty.tools.dotc.Run.compile(Run.scala:78)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:26)
        at dotty.tools.dotc.Driver.process(Driver.scala:124)
        at dotty.tools.dotc.Driver.process(Driver.scala:93)
        at dotty.tools.vulpix.ParallelTesting$Test.compile(ParallelTesting.scala:386)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.$anonfun$checkTestSource$1(ParallelTesting.scala:451)
        at dotty.tools.vulpix.ParallelTesting$Test.tryCompile(ParallelTesting.scala:317)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.checkTestSource(ParallelTesting.scala:449)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run(ParallelTesting.scala:204)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run$(ParallelTesting.scala:203)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.run(ParallelTesting.scala:447)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
java.nio.file.FileSystemException: /Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/Test.tasty: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at dotty.tools.io.AbstractFile.fileOrSubdirectoryNamed(AbstractFile.scala:247)
        at dotty.tools.io.AbstractFile.fileNamed(AbstractFile.scala:258)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile(BytecodeWriters.scala:35)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile$(BytecodeWriters.scala:28)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFile(GenBCode.scala:59)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile(BCodeHelpers.scala:38)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile$(BCodeHelpers.scala:37)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFileForClassfile(GenBCode.scala:59)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.$anonfun$visit$2(GenBCode.scala:209)
        at scala.Option.foreach(Option.scala:257)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.visit(GenBCode.scala:204)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.run(GenBCode.scala:152)
        at dotty.tools.backend.jvm.GenBCodePipeline.buildAndSendToDisk(GenBCode.scala:367)
        at dotty.tools.backend.jvm.GenBCodePipeline.run(GenBCode.scala:333)
        at dotty.tools.backend.jvm.GenBCode.run(GenBCode.scala:54)
        at dotty.tools.dotc.core.Phases$Phase.$anonfun$runOn$1(Phases.scala:289)
        at scala.collection.immutable.List.map(List.scala:283)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:287)
        at dotty.tools.dotc.core.Phases$Phase.runOn$(Phases.scala:286)
        at dotty.tools.backend.jvm.GenBCode.runOn(GenBCode.scala:38)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$3(Run.scala:125)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.trackTime(Stats.scala:47)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2(Run.scala:123)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2$adapted(Run.scala:121)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:121)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$1(Run.scala:141)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:86)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:107)
        at dotty.tools.dotc.Run.compileSources(Run.scala:94)
        at dotty.tools.dotc.Run.compile(Run.scala:78)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:26)
        at dotty.tools.dotc.Driver.process(Driver.scala:124)
        at dotty.tools.dotc.Driver.process(Driver.scala:93)
        at dotty.tools.vulpix.ParallelTesting$Test.compile(ParallelTesting.scala:386)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.$anonfun$checkTestSource$1(ParallelTesting.scala:451)
        at dotty.tools.vulpix.ParallelTesting$Test.tryCompile(ParallelTesting.scala:317)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.checkTestSource(ParallelTesting.scala:449)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run(ParallelTesting.scala:204)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run$(ParallelTesting.scala:203)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.run(ParallelTesting.scala:447)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

Error while emitting t2503.scala
/Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/SB.tasty: Too many open files

Error while emitting t2503.scala
/Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/Test.tasty: Too many open files

error while loading Throwable,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files

error while loading Byte,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files

(And many more)

[allanrenucci]

@AleksanderBG you've encountered something similar. What OS are you running?

[abgruszecki]

@allanrenucci - macOS. In my case, the error appeared after running the entire test suite multiple times in quick succession, so that may have been the cause.

[nicolasstucki]

I have encountered the same issue on Mac OS :(

[odersky]

Same for me on MacOS. I can't even run a single test anymore. It runs out of files in compile/pos. Could it be that some file handles are not closed anymore?

[nicolasstucki]

By bisecting again I found 7ff1cb6 to be the first bad commit.