Fix Packaging For Development Use

[jackkoenig]

A point of ire for many people, see #499 and #551 (and probably others).

While using chisel3, firrtl, chisel-testers, etc. using library dependencies is great, we really need to be able to do it in a way that works well for development use too.

rocket-chip currently builds a firrtl jar and puts it into the unmanaged dependencies so that chisel3 can skip the library dependency. This is kind of broken in the sense that it requires using a Makefile to wrap up multiple calls to sbt.

To the best of my knowledge, sbt does not easily allow you to overwrite library dependencies when a local version (not publish-local) is available (see sbt/sbt#2777). The current behavior of chisel3 that accepts a firrtl.jar in lieu of the library dependency is an example of working around this problem but leaves much to be desired. Related StackOverflow posts 1 and 2.

We have discussed merging all of the related projects into a single sbt library, which has some benefits. It would solve the current issue in rocket-chip (kind of), but does not solve the fundamental problem that will spring up for anyone trying to create a chisel or firrtl library (eg. barstools or rocket-chip as a library).

I have an idea that might work:
Create a convention that all of our sbt projects look for a local checkout of their dependencies in a specifically named directory (for example: chisel3 would look for ./firrtl). If one is found, it uses the local version as a project instead of using library dependency
Potential Problems: I'm not sure if this can be done in a build.sbt but I believe it can be done in a build.scala. It also requires some kind of setup logic (like a script). Confusing error if you forget to run setup script. Related SBT plugin (old)

[donggyukim]

I'm totally fine with handling it in build.scala, but not sure other people are ok with it. What is exactly the setup script? It's also fine if we need to run it only once.

[edwardcwang]

In addition to the specifically named directory, it would be nice to make that directory overriddable by environment variable. That way, say if I have a lot of repos which all have chisel3/firrtl, I can store one copy of chisel3/firrtl and just point all my repos to that one.

[ucbjrl]

I propose a variant of this. We define a Dependencies object in project/Dependencies.scala for each of the BigFour (chisel3, firrtl, firrtl-interpreter, chisel-testers):

import sbt._
import Keys._

object Dependencies {
  // The basic chisel dependencies.
  val chiselDependencies = collection.immutable.HashMap[String, Seq[String]](
    "chisel" -> Seq("firrtl"),
    "chisel-testers" -> Seq("firrtl", "firrtl-interpreter"),
    "firrtl" -> Seq(),
    "firrtl-interpreter" -> Seq("firrtl")
  )

  // The following are the default development versions of chisel libraries.
  val chiselDefaultVersions = Map(
    "chisel" -> "3.1-SNAPSHOT",
    "chisel3" -> "3.1-SNAPSHOT",
    "firrtl" -> "1.1-SNAPSHOT",
    "firrtl-interpreter" -> "1.1-SNAPSHOT",
    "chisel-testers" -> "1.2-SNAPSHOT"
    )

  // Give a module/project name, return the ModuleID
  // Provide a managed dependency on X if -DXVersion="" is supplied on the command line (via JAVA_OPTS).
  private def nameToModuleID(name: String): ModuleID = {
    "edu.berkeley.cs" %% name % sys.props.getOrElse(name + "Version", chiselDefaultVersions(name))
  }

  // For a given chisel project, return a sequence of project references,
  //  suitable for use as an argument to dependsOn().
  def chiselProjectDependencies(name: String): Seq[ClasspathDep[ProjectReference]] = {
    //    (chiselDependencies(name) map {p: String => classpathDependency(subProjects(p))})
    Seq()
  }

  // BigFour dependencies as library dependencies.
  def chiselLibraryDependencies(name: String): Seq[ModuleID] = {
    chiselDependencies(name) map { nameToModuleID(_) }
  }

}

Then the individual build.sbt's contain:

val internalName = "chisel-testers"

libraryDependencies ++= chiselLibraryDependencies(internalName)
  ...

dependsOn((chiselProjectDependencies(internalName)):_*)

A super-project (containing some combination of the BIgFour as subprojects) has in its project/Dependencies.scala:

import sbt._
import Keys._

object Dependencies {
  // The basic chisel dependencies.
  val chiselDependencies = collection.immutable.HashMap[String, Seq[String]](
    "chisel" -> Seq("firrtl"),
    "chisel-testers" -> Seq("firrtl", "firrtl-interpreter"),
    "firrtl" -> Seq(),
    "firrtl-interpreter" -> Seq("firrtl")
  )

  // A map from name (string) to project build definition.
  // These will be used to construct the project dependsOn() dependencies.
  lazy val subProjects = collection.immutable.HashMap[String, ProjectReference](
    "chisel" -> ChiselBuild.chisel,
    "chisel-testers" -> ChiselBuild.chisel_testers,
    "firrtl" -> ChiselBuild.firrtl,
    "firrtl-interpreter" -> ChiselBuild.firrtl_interpreter
  )

  // For a given chisel project, return a sequence of project references,
  //  suitable for use as an argument to dependsOn().
  def chiselProjectDependencies(name: String): Seq[ClasspathDep[ProjectReference]] = {
    (chiselDependencies(name) map {p: String => classpathDependency(subProjects(p))})
  }

  // The following are the default development versions of chisel libraries,
  //  not the "release" versions.
  val chiselDefaultVersions = Map(
    "chisel" -> "3.1-SNAPSHOT",
    "chisel3" -> "3.1-SNAPSHOT",
    "firrtl" -> "1.1-SNAPSHOT",
    "firrtl-interpreter" -> "1.1-SNAPSHOT",
    "chisel-testers" -> "1.2-SNAPSHOT"
    )

  // Give a module/project name, return the ModuleID
  // Provide a managed dependency on X if -DXVersion="" is supplied on the command line (via JAVA_OPTS).
  private def nameToModuleID(name: String): ModuleID = {
    "edu.berkeley.cs" %% name % sys.props.getOrElse(name + "Version", chiselDefaultVersions(name))
  }

  // Since we include the libraries as subprojects,
  //  there aren't any library dependencies, but if there were,
  //  they would be these:
  def chiselLibraryDependencies(name: String): Seq[ModuleID] = {
    // chiselDependencies(name) map { nameToModuleID(_) }
    Seq()
  }

}

The super project also contains a project/ChiselBuild.scala containing:

import sbt._
import Keys._

object ChiselBuild extends Build {

  lazy val commonSettings = Seq (
    organization := "edu.berkeley.cs",
    scalaVersion := "2.11.8",

    resolvers ++= Seq(
      Resolver.sonatypeRepo("snapshots"),
      Resolver.sonatypeRepo("releases")
    ),

    javacOptions ++= Seq("-source", "1.7", "-target", "1.7")
  )

  lazy val publishSettings = Seq (
    publishMavenStyle := true,
    publishArtifact in Test := false,
    pomIncludeRepository := { x => false },

    publishTo <<= version { v: String =>
      val nexus = "https://oss.sonatype.org/"
      if (v.trim.endsWith("SNAPSHOT")) {
	Some("snapshots" at nexus + "content/repositories/snapshots")
      }
      else {
	Some("releases" at nexus + "service/local/staging/deploy/maven2")
      }
    }
  )

  lazy val chisel = (project in file("chisel3")).
    settings(commonSettings: _*).
    settings(publishSettings: _*).
    dependsOn(firrtl)

  lazy val chisel_testers = (project in file("chisel-testers")).
    settings(commonSettings: _*).
    settings(publishSettings: _*).
    dependsOn(chisel, firrtl, firrtl_interpreter)

  lazy val firrtl = (project in file("firrtl")).
    settings(commonSettings: _*).
    settings(publishSettings: _*)

  lazy val firrtl_interpreter = (project in file("firrtl-interpreter")).
    settings(commonSettings: _*).
    settings(publishSettings: _*).
    dependsOn(firrtl)

}

This relies on the fact that (currently) the project/*.scala files are ignored for subprojects.

This could use some tweaking to minimize the amount of code in the project/*.scala files and eliminate duplication between them and the build.sbt, but I'm experimenting with it in my local copy of chisel-release and it seems to work. I can build and test the entire release without touching the .../.ivy2/{cache,local}/edu.berkeley.cs directories.

[jackkoenig]

All the setup script would need to do is add symbolic links in each directory. For example, given directory structure:

rocket-chip
  - chisel3  (depends on firrtl)
  - firrtl
  - barstools  (depends on chisel3 and firrtl)

You would need to create links in the subprojects that have dependencies:

rocket-chip
  - chisel3
    - firrtl -> ../firrtl
  - firrtl
  - barstools
    - chisel3 -> ../chisel3
    - firrtl -> ../firrtl

Unless I am very much mistaken, sbt cannot look at the full project structure at the time it needs to load subprojects, so you either need to use an environment variable or just a local symbolic link. @edwardcwang environment variables could work too.

[jackkoenig]

@ucbjrl This looks interesting, do you have a branch or branches I can try out to understand it a little better?

[ucbjrl]

Check out the deprepkg branch of chisel-release. (Don't forget to git submodule update --init --recursive.)

Then make clean test (or sbt clean test) in the top (chisel-release) project.

[ucbjrl]

I've put together an internal document describing my proposed solution. Once we've reach consensus, we should open this for external comment.

[azidar]

This issue requires a champion to pursue a solution.

See rocket-chip sbt for inspiration (https://github.com/chipsalliance/rocket-chip/blob/master/build.sbt#L60).

Until then, we are closing this issue.