From 7a810d45d182873b1f5dab81b5ca69a1f1c8e571 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubom=C3=ADr=20Bulej?= Date: Wed, 18 Oct 2023 20:02:46 +0200 Subject: [PATCH 1/3] Target Java 11 by default, keep Java 8 for core/harness --- CONTRIBUTING.md | 8 +-- README.md | 70 +++++++++---------- benchmarks/actors-reactors/reactors/build.sbt | 1 + build.sbt | 25 ++++--- project/build.scala | 9 ++- .../harness/MarkdownGenerator.scala | 10 +-- 6 files changed, 67 insertions(+), 56 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 44e4bfc4..0f5549c9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,15 +14,14 @@ the community. ### Setting up your environment -Renaissance Benchmark Suite is built via SBT and is written in both -Java and Scala. +Renaissance Benchmark Suite is built via SBT and is written in both Java and Scala. Several tips on how to setup your IDE are in `documentation/ide.md`. ### Building the suite -To build the suite and create the so-called fat JAR (or super JAR), you only -need to run `sbt` build tool as follows: +To build the suite and create the so-called fat JAR (or super JAR), you need to have +a JDK (at least version 11) installed and run the `sbt` build tool as follows: ``` $ tools/sbt/bin/sbt renaissancePackage @@ -31,7 +30,6 @@ $ tools/sbt/bin/sbt renaissancePackage This will retrieve all the dependencies, compile all the benchmark projects and the harness, bundle the JARs and create the final JAR under the `target` directory. - ### Making a contribution Contribution to Renaissance Benchmark Suite is not only about adding new benchmarks. diff --git a/README.md b/README.md index ce9dcf3a..cbb77982 100644 --- a/README.md +++ b/README.md @@ -21,16 +21,16 @@ from . If you wish to build it yourself, please, consult [CONTRIBUTING.md](CONTRIBUTING.md) for instructions on building. -To run a Renaissance benchmark, you need to have a JRE installed. -This allows you to execute the following `java` command: - +To run a Renaissance benchmark, you need to have a JRE version 11 (or later) +installed and execute the following `java` command: ``` $ java -jar 'renaissance-gpl-0.14.2.jar' ``` -Above, `` is the list of benchmarks that you wish to run. -For example, you can specify `scala-kmeans` as the benchmark. +In the above command, `` is the list of benchmarks that you want to run. +You can refer to individual benchmarks, e.g., `scala-kmeans`, or a group of benchmarks, +e.g., `apache-spark`. The suite generally executes the benchmark's measured operation multiple times. By default, the suite executes each benchmark operation for a specific number of times. The benchmark-specific @@ -67,55 +67,55 @@ The following is the complete list of benchmarks, separated into groups. - `als` - Runs the ALS algorithm from the Spark ML library. \ - Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `chi-square` - Runs the chi-square test from Spark MLlib. \ - Default repetitions: 60; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 60; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `dec-tree` - Runs the Random Forest algorithm from the Spark ML library. \ - Default repetitions: 40; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 40; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `gauss-mix` - Computes a Gaussian mixture model using expectation-maximization. \ - Default repetitions: 40; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 40; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `log-regression` - Runs the Logistic Regression algorithm from the Spark ML library. \ - Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `movie-lens` - Recommends movies using the ALS algorithm. \ - Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `naive-bayes` - Runs the multinomial Naive Bayes algorithm from the Spark ML library. \ - Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `page-rank` - Runs a number of PageRank iterations, using RDDs. \ - Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 11 and later #### concurrency - `akka-uct` - Runs the Unbalanced Cobwebbed Tree actor workload in Akka. \ - Default repetitions: 24; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 24; MIT license, MIT distribution; Supported JVM: 11 and later - `fj-kmeans` - Runs the K-Means algorithm using the fork/join framework. \ - Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `reactors` - Runs benchmarks inspired by the Savina microbenchmark workloads in a sequence on Reactors.IO. \ - Default repetitions: 10; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 10; MIT license, MIT distribution; Supported JVM: 11 and later #### database - `db-shootout` - Executes a shootout test using several in-memory databases. \ - Default repetitions: 16; APACHE2 license, MIT distribution; Supported JVM: 1.8 - 18 + Default repetitions: 16; APACHE2 license, MIT distribution; Supported JVM: 11 - 18 - `neo4j-analytics` - Executes Neo4j graph queries against a movie database. \ @@ -125,55 +125,55 @@ The following is the complete list of benchmarks, separated into groups. - `future-genetic` - Runs a genetic algorithm using the Jenetics library and futures. \ - Default repetitions: 50; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 50; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `mnemonics` - Solves the phone mnemonics problem using JDK streams. \ - Default repetitions: 16; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 16; MIT license, MIT distribution; Supported JVM: 11 and later - `par-mnemonics` - Solves the phone mnemonics problem using parallel JDK streams. \ - Default repetitions: 16; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 16; MIT license, MIT distribution; Supported JVM: 11 and later - `rx-scrabble` - Solves the Scrabble puzzle using the Rx streams. \ - Default repetitions: 80; GPL2 license, GPL3 distribution; Supported JVM: 1.8 and later + Default repetitions: 80; GPL2 license, GPL3 distribution; Supported JVM: 11 and later - `scrabble` - Solves the Scrabble puzzle using JDK Streams. \ - Default repetitions: 50; GPL2 license, GPL3 distribution; Supported JVM: 1.8 and later + Default repetitions: 50; GPL2 license, GPL3 distribution; Supported JVM: 11 and later #### scala - `dotty` - Runs the Dotty compiler on a set of source code files. \ - Default repetitions: 50; BSD3 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 50; BSD3 license, MIT distribution; Supported JVM: 11 and later - `philosophers` - Solves a variant of the dining philosophers problem using ScalaSTM. \ - Default repetitions: 30; BSD3 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 30; BSD3 license, MIT distribution; Supported JVM: 11 and later - `scala-doku` - Solves Sudoku Puzzles using Scala collections. \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later - `scala-kmeans` - Runs the K-Means algorithm using Scala collections. \ - Default repetitions: 50; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 50; MIT license, MIT distribution; Supported JVM: 11 and later - `scala-stm-bench7` - Runs the stmbench7 benchmark using ScalaSTM. \ - Default repetitions: 60; BSD3, GPL2 license, GPL3 distribution; Supported JVM: 1.8 and later + Default repetitions: 60; BSD3, GPL2 license, GPL3 distribution; Supported JVM: 11 and later #### web - `finagle-chirper` - Simulates a microblogging service using Twitter Finagle. \ - Default repetitions: 90; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 90; APACHE2 license, MIT distribution; Supported JVM: 11 and later - `finagle-http` - Sends many small Finagle HTTP requests to a Finagle HTTP server and awaits response. \ - Default repetitions: 12; APACHE2 license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 12; APACHE2 license, MIT distribution; Supported JVM: 11 and later @@ -184,27 +184,27 @@ purposes: - `dummy-empty` - A dummy benchmark which only serves to test the harness. \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later - `dummy-failing` - A dummy benchmark for testing the harness (fails during iteration). \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later - `dummy-param` - A dummy benchmark for testing the harness (test configurable parameters). \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later - `dummy-setup-failing` - A dummy benchmark for testing the harness (fails during setup). \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later - `dummy-teardown-failing` - A dummy benchmark for testing the harness (fails during teardown). \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later - `dummy-validation-failing` - A dummy benchmark for testing the harness (fails during validation). \ - Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 1.8 and later + Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later diff --git a/benchmarks/actors-reactors/reactors/build.sbt b/benchmarks/actors-reactors/reactors/build.sbt index 3d80f28a..eba5df49 100644 --- a/benchmarks/actors-reactors/reactors/build.sbt +++ b/benchmarks/actors-reactors/reactors/build.sbt @@ -14,6 +14,7 @@ def projectSettings(suffix: String) = { scalacOptions := (parentProject / scalacOptions).value ++ Seq( "-feature", "-no-specialization" ), + javacOptions := (parentProject / javacOptions).value, Compile / unmanagedSourceDirectories := Seq( baseDirectory.value / "jvm" / "src" / "main" / "java", baseDirectory.value / "jvm" / "src" / "main" / "scala", diff --git a/build.sbt b/build.sbt index 0ab74d0d..7c808cdd 100644 --- a/build.sbt +++ b/build.sbt @@ -33,7 +33,6 @@ Global / onLoad := { previousOnLoad.andThen(state => "setupPrePush" :: state) } -// // Support for distributions with different licenses. // val nonGplOnly = SettingKey[Boolean]( @@ -56,14 +55,13 @@ ThisBuild / git.useGitDescribe := true // // Compilation settings // -val javaRelease = "8" +val javaRelease = "11" val scalaVersion212 = "2.12.18" val scalaVersion213 = "2.13.12" val scalaVersion3 = "3.3.1" // Explicitly target a specific JDK release. -ThisBuild / javacOptions ++= Seq("-source", javaRelease, "-target", javaRelease) -ThisBuild / scalacOptions ++= Seq("-release", javaRelease) +ThisBuild / javacOptions ++= Seq("--release", javaRelease) // Use common organization name. ThisBuild / organization := "org.renaissance" @@ -78,15 +76,19 @@ lazy val commonSettingsNoScala = Seq( ) lazy val commonSettingsScala212 = Seq( - scalaVersion := scalaVersion212 + scalaVersion := scalaVersion212, + // Scala 2.12 can only emit valid Java 8 classes. + scalacOptions ++= Seq(s"-release:$javaRelease") ) lazy val commonSettingsScala213 = Seq( - scalaVersion := scalaVersion213 + scalaVersion := scalaVersion213, + scalacOptions ++= Seq(s"-release:$javaRelease") ) lazy val commonSettingsScala3 = Seq( scalaVersion := scalaVersion3, + scalacOptions ++= Seq("-java-output-version", javaRelease), dependencyOverrides ++= Seq( // Force common version of Scala 2.13 library. "org.scala-lang" % "scala-library" % scalaVersion213 @@ -173,7 +175,8 @@ lazy val renaissanceCore = (project in file("renaissance-core")) .settings( name := "renaissance-core", commonSettingsNoScala, - Compile / mainClass := Some(launcherMainClass) + Compile / mainClass := Some(launcherMainClass), + Compile / javacOptions := Seq("--release", "8") ) val renaissanceHarnessCommonSettings = Seq( @@ -184,6 +187,7 @@ val renaissanceHarnessCommonSettings = Seq( "io.spray" %% "spray-json" % "1.3.6" ), Compile / mainClass := Some(harnessMainClass), + Compile / javacOptions := Seq("--release", "8"), Compile / packageBin / packageOptions += generateManifestAttributesTask.value ) @@ -191,7 +195,8 @@ lazy val renaissanceHarness3 = (project in file("renaissance-harness")) .settings( name := "renaissance-harness_3", commonSettingsScala3, - renaissanceHarnessCommonSettings + renaissanceHarnessCommonSettings, + scalacOptions := Seq("-java-output-version", "8") ) .dependsOn(renaissanceCore % "provided") @@ -199,7 +204,8 @@ lazy val renaissanceHarness213 = (project in file("renaissance-harness")) .settings( name := "renaissance-harness_2.13", commonSettingsScala213, - renaissanceHarnessCommonSettings + renaissanceHarnessCommonSettings, + scalacOptions := Seq("-release:8") ) .dependsOn(renaissanceCore % "provided") @@ -208,6 +214,7 @@ lazy val renaissanceHarness212 = (project in file("renaissance-harness")) name := "renaissance-harness_2.12", commonSettingsScala212, renaissanceHarnessCommonSettings, + scalacOptions := Seq("-release:8"), libraryDependencies ++= Seq( // Needed to compile Scala 2.13 collections with Scala 2.12. "org.scala-lang.modules" %% "scala-collection-compat" % scalaCollectionCompatVersion diff --git a/project/build.scala b/project/build.scala index 1ae8c3b0..6c5b77a0 100644 --- a/project/build.scala +++ b/project/build.scala @@ -16,6 +16,11 @@ class BenchmarkInfo( val scalaVersion: String ) { + /** + * Default minimal JVM version for benchmarks that do not specify it explicitly. + */ + private val JVM_VERSION_MIN_DEFAULT = "11" + private def kebabCase(s: String): String = { val camelCaseName = if (s.last == '$') s.init else s val pattern = Pattern.compile("([A-Za-z])([A-Z])") @@ -133,8 +138,8 @@ class BenchmarkInfo( } def jvmVersionMin: String = { - // Require at least JVM 1.8 where unspecified. - getAnnotation(classOf[RequiresJvm]).map(_.value()).getOrElse("1.8") + // When unspecified, require the default minimum version. + getAnnotation(classOf[RequiresJvm]).map(_.value()).getOrElse(JVM_VERSION_MIN_DEFAULT) } def jvmVersionMax: String = { diff --git a/renaissance-harness/src/main/scala/org/renaissance/harness/MarkdownGenerator.scala b/renaissance-harness/src/main/scala/org/renaissance/harness/MarkdownGenerator.scala index 768f3a1b..b9d13439 100644 --- a/renaissance-harness/src/main/scala/org/renaissance/harness/MarkdownGenerator.scala +++ b/renaissance-harness/src/main/scala/org/renaissance/harness/MarkdownGenerator.scala @@ -236,16 +236,16 @@ from . If you wish to build it yourself, please, consult [CONTRIBUTING.md](CONTRIBUTING.md) for instructions on building. -To run a Renaissance benchmark, you need to have a JRE installed. -This allows you to execute the following `java` command: - +To run a Renaissance benchmark, you need to have a JRE version 11 (or later) +installed and execute the following `java` command: ``` $$ java -jar 'renaissance-gpl-${tags("renaissanceVersion")}.jar' ``` -Above, `` is the list of benchmarks that you wish to run. -For example, you can specify `scala-kmeans` as the benchmark. +In the above command, `` is the list of benchmarks that you want to run. +You can refer to individual benchmarks, e.g., `scala-kmeans`, or a group of benchmarks, +e.g., `apache-spark`. The suite generally executes the benchmark's measured operation multiple times. By default, the suite executes each benchmark operation for a specific number of times. The benchmark-specific From b40202b101098039ef7ce9dd68ac449139058f0f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubom=C3=ADr=20Bulej?= Date: Thu, 19 Oct 2023 02:00:11 +0200 Subject: [PATCH 2/3] Report failures when creating benchmark execution driver --- .../harness/RenaissanceSuite.scala | 57 ++++++++++++------- 1 file changed, 38 insertions(+), 19 deletions(-) diff --git a/renaissance-harness/src/main/scala/org/renaissance/harness/RenaissanceSuite.scala b/renaissance-harness/src/main/scala/org/renaissance/harness/RenaissanceSuite.scala index 7b6ba908..0eb94ad8 100644 --- a/renaissance-harness/src/main/scala/org/renaissance/harness/RenaissanceSuite.scala +++ b/renaissance-harness/src/main/scala/org/renaissance/harness/RenaissanceSuite.scala @@ -127,7 +127,12 @@ object RenaissanceSuite { val dispatcher = createEventDispatcher(plugins, writers) // Note: no access to Config beyond this point. - runBenchmarks(suite, benchmarks, policy, dispatcher) + val failedBenchmarks = runBenchmarks(suite, benchmarks, policy, dispatcher) + if (failedBenchmarks.nonEmpty) { + val failedBenchmarksList = failedBenchmarks.map(_.name()).mkString(", ") + println(s"The following benchmarks failed: $failedBenchmarksList") + sys.exit(1) + } } } @@ -144,7 +149,7 @@ object RenaissanceSuite { benchmarks: Seq[BenchmarkDescriptor], policy: ExecutionPolicy, dispatcher: EventDispatcher - ): Unit = { + ): Seq[BenchmarkDescriptor] = { // TODO: Why collect failing benchmarks instead of just quitting whenever one fails? val failedBenchmarks = mutable.Buffer[BenchmarkDescriptor]() @@ -153,42 +158,56 @@ object RenaissanceSuite { // Notify observers that the suite is set up. dispatcher.notifyAfterHarnessInit() - try { - for (descriptor <- benchmarks) { - val driver = ExecutionDriver.create(suite, descriptor, dispatcher, policy, vmStartNanos) + for (descriptor <- benchmarks) { + try { + val driver = ExecutionDriver.create( + suite, + descriptor, + dispatcher, + policy, + vmStartNanos + ) try { driver.executeBenchmark() } catch { - case t: Throwable => - // Notify observers that a benchmark failed. + case cause: Throwable => + // Notify observers that a benchmark failed, because they + // have been notified about the benchmark setup phase. dispatcher.notifyOnBenchmarkFailure(descriptor.name) failedBenchmarks += descriptor - t match { + cause match { case _: ValidationException => Console.err.println( - s"Benchmark '${descriptor.name()}' failed result validation:\n${t.getMessage}" + s"Benchmark '${descriptor.name()}' failed result validation:\n${cause.getMessage}" ) case _ => - Console.err.println(s"Benchmark '${descriptor.name()}' failed with exception:") - t.printStackTrace(Console.err) + Console.err.println( + s"Benchmark '${descriptor.name()}' failed with exception:" + ) + cause.printStackTrace(Console.err) } } - } - } finally { - // Notify listeners that the suite is shutting down. - dispatcher.notifyBeforeHarnessShutdown() + } catch { + case cause: Throwable => + // Observers are not notified if a benchmark failed to load, + // because they do not know about the benchmark at all. + failedBenchmarks += descriptor - if (failedBenchmarks.nonEmpty) { - val failedBenchmarksList = failedBenchmarks.map(_.name()).mkString(", ") - println(s"The following benchmarks failed: $failedBenchmarksList") - sys.exit(1) + Console.err.println( + s"Failed to load benchmark '${descriptor.name()}': ${cause.getMessage}" + ) } } + + // Notify listeners that the suite is shutting down. + dispatcher.notifyBeforeHarnessShutdown() + + failedBenchmarks.toSeq } private def getVmStartNanos = { From e0c4619618896cbaa12a28bce2f4c88c42d1e4a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lubom=C3=ADr=20Bulej?= Date: Sun, 22 Oct 2023 20:22:09 +0200 Subject: [PATCH 3/3] Use play-json 2.10.1 again now that we require Java 11 --- build.sbt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.sbt b/build.sbt index 7c808cdd..1a025309 100644 --- a/build.sbt +++ b/build.sbt @@ -362,7 +362,7 @@ lazy val neo4jBenchmarks = (project in file("benchmarks/neo4j")) // neo4j 5.x supports Scala 2.13 and requires JDK17. "org.neo4j" % "neo4j" % "5.12.0", // play-json 2.10.x requires SBT running on JDK11 to compile. - "com.typesafe.play" %% "play-json" % "2.9.4" + "com.typesafe.play" %% "play-json" % "2.10.1" ), excludeDependencies ++= Seq( // Drop dependencies that are not really used by the benchmark.