wip: fix scripted tests #1

jtjeferreira · 2024-12-04T22:34:34Z

Work in progress to fix scripted tests. Do you prefer to discuss here or in sbt#1647?

The biggest decision to make is what should target be? Keep backwards compatibility with ./target/universal? Use the new default target ( ./target/out/jvm/scala-3.3.3/universal )? Or something else like ./target/out/universal?

…sbt 2.x and project matrix target is target/out/jvm/scala-3.3.3/ alternatively we could change the Universal plugin so that config / target := baseDirectory.value / "target" / config.name

…tion

…ompilation

…ave javafx)

…doing

eed3si9n · 2024-12-05T02:31:07Z

Work in progress to fix scripted tests. Do you prefer to discuss here or in sbt#1647?

Either is fine. Thanks for this! I haven't done it yet, but I was thinking sbt should add some special scripted test command in the next 2.x version like exists-in-target or something, but hardcoding the target works for me too.

jtjeferreira · 2024-12-08T18:45:42Z

windows tests are faling with the error below because of sbt/sbt@d0003bb

Error:  java.nio.file.InvalidPathException: Illegal char <:> at index 90: C:\Users\runneradmin\AppData\Local\Temp\sbt_540f99c3\project\project\target\config-classes:C:\Users\RUNNER~1\AppData\Local\Temp\sbt_540f99c3\global\boot\scala-3.3.3\org.scala-sbt\sbt\2.0.0-M2\actions_3-2.0.0-M2.jar: ....
Error:  	at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
Error:  	at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
Error:  	at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
Error:  	at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
Error:  	at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
Error:  	at java.nio.file.Paths.get(Paths.java:84)
Error:  	at dotty.tools.dotc.core.MacroClassLoader$.$anonfun$1(MacroClassLoader.scala:25)
Error:  	at scala.collection.immutable.List.map(List.scala:246)
Error:  	at dotty.tools.dotc.core.MacroClassLoader$.makeMacroClassLoader(MacroClassLoader.scala:25)
Error:  	at dotty.tools.dotc.core.MacroClassLoader$.init(MacroClassLoader.scala:19)
Error:  	at dotty.tools.dotc.Driver.setup(Driver.scala:79)
Error:  	at sbt.internal.Eval$EvalDriver.<init>(Eval.scala:55)
Error:  	at sbt.internal.Eval.driver$lzyINIT1(Eval.scala:47)
Error:  	at sbt.internal.Eval.driver(Eval.scala:47)
Error:  	at sbt.internal.Eval.compileAndLoad(Eval.scala:233)
Error:  	at sbt.internal.Eval.evalCommon(Eval.scala:214)
Error:  	at sbt.internal.Eval.eval(Eval.scala:121)

jtjeferreira · 2024-12-08T18:58:24Z

the commits setting scalaVersion can be revert when sbt/sbt#7725 is fixed

jtjeferreira · 2024-12-08T19:26:35Z

The biggest decision to make is what should target be? Keep backwards compatibility with ./target/universal? Use the new default target ( ./target/out/jvm/scala-3.3.3/universal )? Or something else like ./target/out/universal?

After thinking for this a while, I think we should use the new default target ( ./target/out/jvm/scala-3.3.3/universal ). It makes sense that sbt native packager plugin uses whatever the target key is defined to. It also makes sense that if one runs +stage, the results are in different directories (./target/out/jvm/scala-3.3.3/universal , ./target/out/jvm/scala-2.13/universal, ./target/out/js/scala-3.3.3/universal, etc). The only downside is that existing scripts that might depend on ./target/universal might have to be rewritten (like what happened in the scripted tests)

I haven't done it yet, but I was thinking sbt should add some special scripted test command in the next 2.x version like exists-in-target or something, but hardcoding the target works for me too.

I think that is a good idea

jtjeferreira · 2024-12-08T19:34:28Z

src/sbt-test/docker/build-command/build.sbt

-import NativePackagerHelper._
-Docker / mappings ++= directory("src/main/resources/docker-test")


@eed3si9n should sbt.io.Mapper#directory and other methods in sbt.io.Mapper that return mappings (i.e (File, String)) be changed to return ( (HashedVirtualFileRef, String) ) so it aligns with the changes to sbt.Keys#mappings which changed from using File to HashedVirtualFileRef?

That makes sense.

unfortunately sbt.io.Mapper is in sbt/io which does not have access to HashedVirtualFileRef...

WDYT about reverting mappings to (File, String)? I gave it a try here

File simultaneously is specific to user's environment, and not specific enough about the content, so it's inappropriate to participate in remote cache. For anything useful, mappings is going to be a critical input, so we have to make this change if we want remote caching to be a major feature of sbt 2.x. more details here
https://eed3si9n.com/sbt-remote-cache-with-bazel-compat/

sbt/sbt for now?

sbt/sbt for now?

yes, but which package/class, so that is available in build.sbt files without import?

We can put in almost any place that makes sense for build.sbt. The imports are https://github.com/sbt/sbt/blob/develop/main/src/main/scala/sbt/internal/BuildUtil.scala#L93-L100, and we control that with https://github.com/sbt/sbt/blob/develop/sbt-app/src/main/scala/sbt/Import.scala.

We can create Mapper in sbt package and put it in https://github.com/sbt/sbt/tree/develop/main-actions/src/main/scala/sbt for example, and remove the Mapper alias in https://github.com/sbt/sbt/blob/ebee3a86d900758b7085bb40979124ad2a9e4587/sbt-app/src/main/scala/sbt/Import.scala#L57

Thanks. I will give it a try later today

@eed3si9n I tried this in sbt/sbt#7939 but I don't think this is enough.

File simultaneously is specific to user's environment, and not specific enough about the content, so it's inappropriate to participate in remote cache. For anything useful, mappings is going to be a critical input, so we have to make this change if we want remote caching to be a major feature of sbt 2.x. more details here
eed3si9n.com/sbt-remote-cache-with-bazel-compat

I understand these concerns, but making user code so verbose will make adoption of sbt 2.x a pain as it is a downgrade in quality from sbt 1.x

WDYT about reverting mappings to (File, String)?

cant we keep mappings key as (File, String), but move the cache implementation details to the task implementation? Even if it make task implementation more verbose?

….x because with sbt 2.x and project matrix target is target/out/jvm/scala-3.3.3/

… is scala 3

jtjeferreira · 2024-12-20T17:31:19Z

@eed3si9n as you can see here, the only failing test is related to this issue. The ConcurrentModificationException
issue is still a problem, but it usually goes away after a retry of the job...

eed3si9n · 2024-12-20T17:36:31Z

Yea. It turns out Retry(...) only does IOExceptions, and what's failing is concurrent something.

eed3si9n · 2024-12-26T23:29:05Z

I'm going to merge this as-is.

jtjeferreira added 4 commits December 4, 2024 22:39

Workaround: set target folder to what it was in sbt 1.x because with …

79278d5

…sbt 2.x and project matrix target is target/out/jvm/scala-3.3.3/ alternatively we could change the Universal plugin so that config / target := baseDirectory.value / "target" / config.name

fix universal scripted tests using PluginCompat functionality

0a6792d

Set scalaVersion manually in scripted jar tests

501a776

fix jar scripted tests by setting target key and fixing scala compila…

fe988bc

…tion

jtjeferreira force-pushed the wip/cross-fix-tests branch from 68950b1 to 37b4c43 Compare December 4, 2024 22:40

jtjeferreira added 3 commits December 5, 2024 00:02

WIP: fix bash scripted tests by setting target key and fixing scala c…

f2101f1

…ompilation

WIP: fix ash scripted tests by setting target key

0696636

fix ash/bash top-level-main scripted test

cd60412

jtjeferreira force-pushed the wip/cross-fix-tests branch from 37b4c43 to 3992421 Compare December 5, 2024 00:02

jtjeferreira added 7 commits December 5, 2024 00:36

fix rpm tests (hopefully, because they fail in macos)

22f6e02

fix debian tests (hopefully, because some fail in macos)

24d68e5

fix jlink tests

0f5004c

fix jdkpackager tests (hopefully, because they fail because I don't h…

f5438d3

…ave javafx)

fix universal/multiproject-classifiers but nit really sure what I am …

8aa0c38

…doing

Set scalaVersion manually in scripted jdkpackager tests

8e16c4d

Set scalaVersion manually in scripted docker tests

70564db

jtjeferreira force-pushed the wip/cross-fix-tests branch from 3992421 to 70564db Compare December 5, 2024 00:39

jtjeferreira mentioned this pull request Dec 5, 2024

Cross build to sbt 2.0.0-M3 sbt/sbt-native-packager#1647

Merged

jtjeferreira mentioned this pull request Dec 6, 2024

[2.x] ConcurrentModificationException in SbtParser sbt/sbt#7873

Closed

jtjeferreira added 3 commits December 6, 2024 19:37

fix docker tests

7f91d98

fix jdkpackager tests

15d685a

set scalaVersion in graalvm and jdkpackager tests

7c6c77c

fix graalvm-native-image tests

26e9eb3

jtjeferreira commented Dec 8, 2024

View reviewed changes

jtjeferreira force-pushed the wip/cross-fix-tests branch from 16bc1b4 to 8709c75 Compare December 19, 2024 22:57

2.0.0-M3 and fix scala compilation

64a813b

jtjeferreira force-pushed the wip/cross-fix-tests branch from 8709c75 to 64a813b Compare December 19, 2024 23:02

jtjeferreira added 3 commits December 19, 2024 23:30

revert scalaversion dotty scaladoc workaround

341b6d7

[Windows tests] Workaround: set target folder to what it was in sbt 1…

6ebfccf

….x because with sbt 2.x and project matrix target is target/out/jvm/scala-3.3.3/

fix windows tests (hopefully)

8a4743c

jtjeferreira force-pushed the wip/cross-fix-tests branch from ca80d9c to b236cdc Compare December 20, 2024 11:46

set some required scalaVersions, because now the default scalaVersion…

182b814

… is scala 3

jtjeferreira force-pushed the wip/cross-fix-tests branch from b236cdc to 182b814 Compare December 20, 2024 13:23

Merge branch 'wip/cross' into wip/cross-fix-tests

7eb1221

eed3si9n marked this pull request as ready for review December 26, 2024 23:29

eed3si9n merged commit d6d2702 into eed3si9n:wip/cross Dec 26, 2024
11 of 14 checks passed

eed3si9n mentioned this pull request Jan 5, 2025

Add sbt 2.0 crossbuilds sbt/sbt-native-packager#1638

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wip: fix scripted tests #1

wip: fix scripted tests #1

jtjeferreira commented Dec 4, 2024 •

edited

Loading

eed3si9n commented Dec 5, 2024

jtjeferreira commented Dec 8, 2024 •

edited

Loading

jtjeferreira commented Dec 8, 2024

jtjeferreira commented Dec 8, 2024

jtjeferreira Dec 8, 2024

eed3si9n Dec 8, 2024

jtjeferreira Dec 9, 2024

jtjeferreira Dec 9, 2024

eed3si9n Dec 10, 2024

eed3si9n Dec 10, 2024

jtjeferreira Dec 10, 2024

eed3si9n Dec 10, 2024

jtjeferreira Dec 10, 2024

jtjeferreira Dec 10, 2024

jtjeferreira commented Dec 20, 2024

eed3si9n commented Dec 20, 2024

eed3si9n commented Dec 26, 2024

		import NativePackagerHelper._
		Docker / mappings ++= directory("src/main/resources/docker-test")

wip: fix scripted tests #1

wip: fix scripted tests #1

Conversation

jtjeferreira commented Dec 4, 2024 • edited Loading

eed3si9n commented Dec 5, 2024

jtjeferreira commented Dec 8, 2024 • edited Loading

jtjeferreira commented Dec 8, 2024

jtjeferreira commented Dec 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtjeferreira commented Dec 20, 2024

eed3si9n commented Dec 20, 2024

eed3si9n commented Dec 26, 2024

jtjeferreira commented Dec 4, 2024 •

edited

Loading

jtjeferreira commented Dec 8, 2024 •

edited

Loading