Skip to content

Commit

Permalink
Add MonoHashBuilder for staging the params via flow setters
Browse files Browse the repository at this point in the history
Make MonoHash class abstract with static forwarding functions to MonoHashBuilder
Add NoopLogger as default logger in MonoHashBuilder

Inject current MonoHash version into monohash.properties
Provide defaults through param.Config class

Move CmdLineParser and relevant parameters into the param package
Make Concurrency a first-class citizen with two modes: fixed and CPU-relative
Move Logger.Level into a standalone enum in param.LogLevel

Exclude tests in .monohash plan

Bump SBT to 1.4.5
  • Loading branch information
melezov committed Dec 22, 2020
1 parent 7d229a3 commit 4f65429
Show file tree
Hide file tree
Showing 48 changed files with 2,075 additions and 856 deletions.
3 changes: 3 additions & 0 deletions .monohash
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
!.git/
!.gitignore

# Exclude tests
!src/test/

# Exclude docs
!LICENSE
!README.md
Expand Down
39 changes: 17 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,20 @@ runtime code and test code - this means that your CI can reuse a cached build if
run the test part of that CI pipeline while keeping the previously built main artifacts.

MonoHash is **fast**. Running on a cold JVM via the `java -jar monohash.jar` on the
[Linux repository](https://github.com/torvalds/linux/releases/tag/v5.10-rc5) completes in under a second when running on
[Linux repository](https://github.com/torvalds/linux/releases/tag/v5.10) completes in under a second when running on
[Hetzner's PX line](https://www.hetzner.com/dedicated-rootserver/px62-nvme):
```
[melezov@ci-01 monohash]$ sbt package
[info] welcome to sbt 1.4.4 (AdoptOpenJDK Java 1.8.0_272)
[info] welcome to sbt 1.4.5 (AdoptOpenJDK Java 1.8.0_275)
[info] set current project to monohash (in build file:/home/melezov/monohash/)
[info] compiling 20 Java sources to /home/melezov/monohash/target/classes ...
[success] Total time: 2 s, completed Nov 30, 2020 12:39:43 AM
[info] compiling 26 Java sources to /home/melezov/monohash/target/classes ...
[success] Total time: 2 s, completed Dec 22, 2020 7:35:50 PM
[melezov@ci-01 monohash]$ java -jar target/monohash-0.8.1-SNAPSHOT.jar ~/linux-5.10-rc5/
[info] Using [hash plan directory]: '/home/melezov/linux-5.10-rc5/' ...
[info] Hashed 74,822 files with a total of 989,045,424 bytes in 0.792 sec (average speed: 94,472 files/sec, 1,190 MiB/sec)
[info] Executed hash plan by hashing 74,822 files [9be032657c49009add50c4fbccd1d6bffaa4d593] (in 0.917 sec)
[melezov@ci-01 monohash]$ java -jar target/monohash-0.9.0-SNAPSHOT.jar ../linux/linux-5.10/
[info] Using [hash plan directory]: '/home/melezov/linux/linux-5.10/' ...
[info] Hashed 74,825 files with a total of 989,153,287 bytes in 0.815 sec (average speed: 91,809 files/sec, 1,157 MiB/sec)
[info] Executed hash plan by hashing 74,825 files: [3fe808a7fb13acffc3dee02050fbe9fd25230809] (in 0.933 sec)
3fe808a7fb13acffc3dee02050fbe9fd25230809
```


Expand Down Expand Up @@ -154,7 +155,6 @@ If you don't care about programmatic (library) access, you can simply
and use it on the command line.

Running it on the command line allows for some configuration such as choosing the hashing algorithm, concurrency and log levels:

```
Usage: java -jar monohash.jar <options> [hash plan file] [export file (optional)]
Expand All @@ -169,21 +169,16 @@ Options:
The optional `[export file]` argument can be used to dump the hashes for each traversed file.
Running MonoHash on its own folder produces the following export file:
```
81c58d80a333caf6c8e077e809f52755c4bf7b0d .monohash
e585655a90fec41c2b35ed47e14443d25571140b .monohash
1a98da71f224d591991ca4a13663c381a404eede .travis.yml
e91e0a6e2ae28daabd36016589464890e988418e build.sbt
aac1c402f5ba623eefcdf2f1c60258defff1973f project/build.properties
6abd00785fa263b088b1cb1457270418a14fc5bf project/plugins.sbt
bc5c5f56be234577c861a00de3b9a6b52bcd58e4 project/publish.sbt.disabled
8be7a61b089e8250049887b3954037fb8833b13a publish.sbt.disabled
229bbaf4041633f809bf3784a174e7f9c6fc9589 src/main/java/com/oradian/infra/monohash/Algorithm.java
ae0181ec4c07e7d77789d8a66f4d4a2490ebe226 src/main/java/com/oradian/infra/monohash/CmdLineParser.java
81197abdab8214610c2dab552dd4ceb1cdfaeeda src/main/java/com/oradian/infra/monohash/ExitException.java
818b9e83e0a8e2cfbfc5783d28697957be9c8a52 src/main/java/com/oradian/infra/monohash/ExportParsingException.java
b9ab413b5242698be57a2c5c947731d724ee4488 src/main/java/com/oradian/infra/monohash/HashPlan.java
659f65aac9e6054ef762fa3d955311749f883421 src/main/java/com/oradian/infra/monohash/HashResults.java
44b3176c9f64dfeaea81ddccc46006b9dc7e737a src/main/java/com/oradian/infra/monohash/HashWorker.java
75ab95fcec1cd3592c93e4a08fa208d01390f2cc build.sbt
53f4b08ecc75560114eb52cc83670aadbfceec99 project/PropertiesVersion.scala
a12240fc7b0923db1d55e351e6f490b0d501ab5d project/build.properties
8c8e54954cf7ad27c8cd465fabd651d33d5fdfd7 project/build.sbt
...
6389ba9ef4f64dda2296e60dc5aeae351e83a7ea src/main/java/com/oradian/infra/monohash/util/Hex.java
2e5f7c7b14c786be2930b7edc21ade5c2ab9de62 src/main/resources/com/oradian/infra/monohash/param/monohash.properties
900ff2fa3c861780df6ec38d0acbcb8f654a73bf version.sbt
```

The export file's checksum is the actual output from MonoHash. The file paths are sorted alphabetically and are absolute
Expand Down
2 changes: 1 addition & 1 deletion benchmark/project/SbtLogger.scala
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ class SbtLogger(underlying: sbt.Logger, level: Level.Value) extends Logger {
override def warn(msg: String): Unit = underlying.warn(msg)
override def info(msg: String): Unit = underlying.info(msg)
override def debug(msg: String): Unit = underlying.debug(msg)
override def trace(msg: String): Unit = underlying.verbose(msg) // alias for debug, not a real level
override def trace(msg: String): Unit = underlying.verbose(msg) // alias for debug, not a real Level in SBT
}
2 changes: 1 addition & 1 deletion benchmark/project/build.properties
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sbt.version=1.4.4
sbt.version=1.4.5
37 changes: 33 additions & 4 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,18 @@ organization := "com.oradian.infra"
name := "monohash"

libraryDependencies ++= Seq(
"org.specs2" %% "specs2-core" % "4.10.5" % Test,
"org.specs2" %% "specs2-core" % "4.10.5" % Test,
"org.bouncycastle" % "bcprov-jdk15on" % "1.67" % Test,
)

crossPaths := false
autoScalaLibrary := false

javacOptions in doc := Seq(
doc / javacOptions := Seq(
"-encoding", "UTF-8",
"-source", "8",
)
javacOptions := (javacOptions in doc).value ++ (Seq(
javacOptions := (doc / javacOptions).value ++ (Seq(
"-deprecation",
"-parameters",
"-target", "8",
Expand All @@ -32,11 +33,39 @@ scalacOptions := Seq(
"-Yrangepos",
)

fork in Test := true
Test / fork := true
Test / parallelExecution := false
Test / testForkedParallel := false
Test / testGrouping := {
val opts = ForkOptions(
javaHome = (Test / javaHome).value,
outputStrategy = (Test / outputStrategy).value,
bootJars = Vector.empty,
workingDirectory = Some((Test / baseDirectory).value),
runJVMOptions = (Test / javaOptions).value.toVector,
connectInput = (Test / connectInput).value,
envVars = (Test / envVars).value,
)
// run each test in separate forked JVM so that we can e.g.
// - screw up Security Providers
// - test loading of corrupted monohash.properties
// - fiddle with Singletons via reflection ...
(Test / definedTests).value map { test =>
Tests.Group(test.name, Seq(test), Tests.SubProcess(opts))
}
}

jacocoReportSettings := JacocoReportSettings(
// output to HTML for humans, and XML for Codecov
formats = Seq(JacocoReportFormats.ScalaHTML, JacocoReportFormats.XML),
)

Global / onLoad := { state =>
val propertiesFile = baseDirectory.value /
"src" / "main" / "resources" /
"com" / "oradian" / "infra" / "monohash" / "param" / "monohash.properties"
PropertiesVersion.update(sLog.value, propertiesFile, version.value)
state
}

Global / onChangedBuildSource := ReloadOnSourceChanges
36 changes: 36 additions & 0 deletions project/PropertiesVersion.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import java.io._
import java.nio.charset.StandardCharsets
import java.nio.file.Files
import java.util.{Arrays => JArrays}

import nu.studer.java.util.OrderedProperties

object PropertiesVersion {
/** Update the .properties file in the resources with latest version
* and also do a roundtrip through java.util.Properties compatible
* implementation to avoid encoding / escaping gotchas - for there are plenty */
def update(logger: sbt.Logger, propertiesFile: File, version: String): Unit = {
val currentPropsBytes = Files.readAllBytes(propertiesFile.toPath)
val props = {
val tmp = new OrderedProperties.OrderedPropertiesBuilder()
.withSuppressDateInComment(true)
.build()
tmp.load(new ByteArrayInputStream(currentPropsBytes))
tmp
}

props.setProperty("Version", version)
val updatedPropsBytes = {
val baos = new ByteArrayOutputStream
props.store(baos, null)
val body = new String(baos.toByteArray, StandardCharsets.ISO_8859_1)
body.replace("\r", "").getBytes(StandardCharsets.ISO_8859_1)
}

val needsUpdate = !JArrays.equals(currentPropsBytes, updatedPropsBytes)
if (needsUpdate) {
logger.info("Updated: " + propertiesFile)
Files.write(propertiesFile.toPath, updatedPropsBytes)
}
}
}
2 changes: 1 addition & 1 deletion project/build.properties
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sbt.version=1.4.4
sbt.version=1.4.5
9 changes: 9 additions & 0 deletions project/build.sbt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
scalaVersion := "2.12.12"
scalacOptions := Seq(
"-deprecation",
"-encoding", "UTF-8",
"-language:_",
"-unchecked",
)

libraryDependencies += "nu.studer" % "java-ordered-properties" % "1.0.4"
Loading

0 comments on commit 4f65429

Please sign in to comment.