Understanding build targets in scala-cli

[ckipp01]

I went down a bit of a rabbit hole this morning with this and I don't fully understand how scala-cli is creating build targets or what determines a build target. I'll illustrate this with a project I'm currently working on.

Project layout

All the files belong to one project, which is entered through Main.scala.

❯ tree -a .
.
├── .scalafmt.conf
├── DependencySegment.scala
├── Finder.scala
├── Main.scala
└── Repositories.scala

First run

When I run this project for the first time (scala-cli run .) I'll get the dirs that I'd expect to be created (.bsp/ and .scala/), and they have the following in them:

❯ tree -aL 4
.
├── .bsp
│  └── scala-cli.json
├── .scala
│  ├── .bloop
│  │  ├── project_940fb43dce
│  │  │  ├── bloop-internal-classes
│  │  │  └── project_940fb43dce-analysis.bin
│  │  ├── project_940fb43dce.json
│  │  └── project_eaa64efb2b-test.json
│  ├── ide-options.json
│  ├── project_940fb43dce
│  │  └── classes
│  │     └── main
│  └── project_eaa64efb2b
│     └── classes
│        └── test

This somewhat makes sense to me and I can see that there are 2 build targets created project_940fb43dce and project_eaa64efb2b. I assume one is a normal one and the other is a test target based on what is in the classes dir and this seems to be backed up by whats in the bloop dir. This brings me to my first question.

1. How come two unique hashes instead of hash and hast-test to better understand how they correlate?

Opening the project with Metals

We'll now just open up Main.scala with Metals, and this is where things start to get pretty murky for me.

❯ tree -aL 3
.
├── .bloop
│  ├── project_940fb43dce
│  │  ├── bloop-internal-classes
│  │  └── project_940fb43dce-analysis.bin
│  ├── project_940fb43dce.json
│  ├── project_d476da5045
│  │  ├── bloop-internal-classes
│  │  └── project_d476da5045-analysis.bin
│  ├── project_d476da5045-test.json
│  ├── project_d476da5045.json
│  └── project_eaa64efb2b-test.json
├── ide-options.json
├── project_940fb43dce
│  └── classes
│     └── main
├── project_d476da5045
│  └── classes
│     ├── main
│     └── test
└── project_eaa64efb2b
   └── classes
      └── test

We now have a new build target introduced project_d476da5045.

2. Why is the new target necessary here?

Also this new target seems to be represented a bit differently having both main and test contained in it, and that's also reflected in .bloop since we now have the hash and hash-test for this new target. The only difference I can tell is this new one is producing semanticdb and the other one isn't?

3. Why does the pattern seem different for this target with the hash being the same and appending test in bloop for the target and not having a unique has for the target?

Duplication

The final question I have is more about duplication. At this point now we have all of the classes for the project duplicated in a fair amount of places. For example the original run created the classes in .bloop/<hash>/bloop-internal-classes/<another-hash>/ and also .scala/<hash>/main/. If these are created on every run, what are the ones in .scala/<hash>/main/ used for that the ones created in the bloop dir can't be re-used for? When we open Metals this new target also gets all the classes + semanticdb created, however again that semanticdb is already created and located in the .bloop dir.

4. Is all this duplication needed? Can any of it be reduced?

Again, these are all just questions I came up with as exploring that I'm curious about. I'm assuming you'll have some insight that will probably clear it all up for me. Thanks!

[lwronski]

1. Right, it should be only one hash, the another build target is not necessary.
   The question is, why it is working in scala-cli? so, scala-cli add main classes to classPath in test scope, therefore is visible scope main in test.
2. Scala-cli generate semantic DBs for Metals, so the compiler options are slightly different. Therefore, a new build target is necessary.
3. I think, that can be a some kind of bug, because scala-cli should generate only one build target which contains main and test scope.

[ckipp01]

Gotcha, thanks for the extra insight. One other thing I just ran into that I don't fully understand is that I have the following structure:

├── .scalafmt.conf
├── DependencySegment.scala
├── Fetcher.scala
├── Finder.scala
├── Main.scala
├── Metadata.scala
├── Repository.scala
└── tests
   └── Main.test.scala

If I understand the docs correctly, this should be fine and my test source should be Main.test.scala. This allows me to both scala-cli run . and scala-cli test .. However in this current structure when I do test I'm always met with a warning about a missing project.

❯ scli test .
Compiling project (Scala 3.1.0, JVM)
Warning: Flag -deprecation set repeatedly
Warning: Flag -feature set repeatedly
Warning: Flag -explain set repeatedly
Compiled project (Scala 3.1.0, JVM)
Warning: Missing project 'project_90499f3899' may cause compilation issues in project 'project_90499f3899-test'
Compiling project (test, Scala 3.1.0, JVM)
Compiled project (test, Scala 3.1.0, JVM)
io.kipp.exists.MainTests:
  + creds 0.007s
  + bad-creds 0.0s
  + base 0.013s
  + snapshots 0.003s
  + repo-creds 0.018s
  + invalid 0.001s

You can actually see the repo for this here. You can just clone it down and scala-cli test src/. What's going on in this scenario and why does it always warn that there is a missing project?

[lwronski]

It's the bug in the Scala CLI, this warning shouldn't be displayed. As you point out in the first point, scala-cli generate two different hashes for test and main scope.

Let's assume, that we have generated a build test project with the name project_eaa64efb2b-test. And the bug is, that Scala CLI set incorrect dependencies argument in bloop project config. Scala CLI strip suffix '-test' and add project name to dependencies

"dependencies":[
   "project_eaa64efb2b"
],

but this project doesn't exists, because scala-cli generated another hash for main scope.

I'm trying to fix it in #550