runIvyDeps are currently not transitive. Should they be?

[megri]

Let's say we have the modules A <- B <- C where <- denotes a "depends on" relationship

// running with mill version 0.11.12
import mill._, mill.scalalib._

trait BaseModule extends ScalaModule {
  def scalaVersion = "3.5.1"
}

object C extends BaseModule {
  def ivyDeps = Agg(
    ivy"org.slf4j:slf4j-api:2.0.16"
  )
}

object B extends BaseModule {
  def moduleDeps = Seq(C)

  def runIvyDeps = Agg(
    ivy"ch.qos.logback:logback-classic:1.5.10"
  )
}

object A extends BaseModule {
  def moduleDeps = Seq(B)
}

So C introduces slf4j and B a logging implementation. mill A.ivyDepsTree --withRuntime shows

[build.sc] [54/59] compile 
[info] compiling 1 Scala source to /Users/martin/dev/mill-deps-test/out/mill-build/compile.dest/classes ...
[info] done compiling
[26/26] A.ivyDepsTree 
├─ org.scala-lang:scala3-library_3:3.5.1
│  └─ org.scala-lang:scala-library:2.13.14
└─ org.slf4j:slf4j-api:2.0.16

While the example may seem a little contrived—why not define the runtime dependency in the A module?—this has consequences with for instance assemblies. At work we have a project of several Spark applications (as separate modules) that are packaged into one single assembly for easy shipping. These modules have different but non-conflicting [runtime] dependencies, and the assembly won't collect the runtime parts without overriding for instance transitiveIvyDeps (which solves one level of transitivity—it has to be overridden in all modules to include "deep" runIvyDeps.)

Aside from backwards compatibility, would making runIvyDeps transitive be an issue?

[lihaoyi]

IIRC one goal was that artifacts published to maven central and used via ivyDeps should behave the same as artifacts used locally via moduleDeps. The issue is that Maven Central doesn't have "transitive runtime dependencies" in its POM data model, so we can't add such a thing to Mill's local moduleDeps without breaking that equivalence.

We probably aren't 100% strict in adhering to that equivalence relationship, but IIRC that was one of the considerations when making it and compileIvyDeps non-transitive

[megri]

Let's see if I understand this correctly: You're saying that a published maven artefact does not disambiguate between runtime and compile time dependencies in the .pom-file, right?

From what I can see, runtime dependencies in a maven multi-module project are transitive; and I would assume they're included in published artefacts which are meant to be run as-is.

Still, provided above assumptions are correct, I don't really see the problem. When running mill x.assembly the intent is to produce a runnable jar, so it seems logical to include runtime dependencies.

I believe compile-time dependencies in Mill—which as I understand it correspond to the "provided" scope in Maven—are handled correctly; eg. they're not transitive.

[lefou]

Yeah, I also think runtime-dependencies are transitive in Maven. And to an extend also test-scoped dependencies are (and include the runtime scope), but we should safely ignore the test-scope.

[lefou]

Making runIvyDeps transitive in Mill is probably the right thing to do, since we want to handle ivy-dependencies and module-dependencies consistently.

[lihaoyi]

I think that sounds reasonable. I wasn't aware maven had runtime dependencies that were transitive

[lefou]

• I opened runIvyDeps should be transitive for consistency with Maven resolution #3761

[megri]

Related to this, from what I understand maven allows you to specify the scope of module dependencies, which is why they have the matrix in their documentation. Basically a <scope>runtime</scope> dependency pulls the compile and runtime classpath of the downstream module into the runtime scope of the dependent module.

AFAIK Mill doesn't do anything like that and I'm not even sure it's a good idea, but is it something that has been considered?

[lefou]

I guess wee need to introduce a runModuleDeps as a dual to runIvyDeps.

[lefou]

So, whenever you had the need to add something to runIvyDeps but you want to implement it in the same project, you need to use runModuleDeps.

[megri]

If we want to conform to the Maven scoping model, but I'm not sure we want or need to. A already brings in B's ivyDeps through the module relationship, same as in Maven, but unlike Maven B's runIvyDeps are left out. We'd only need a runModuleDeps if we want to put all dependencies from B in A's runtime classpath — so effectively hiding local packages and such defined in B from IDEs.

[megri]

… which, I guess, could be a desirable thing for external modules 🤔.. perhaps.