Allow users to export and import coursier caches…

[alexarchambault]

…so that users with limited internet connections (like can happen in some corporate environments) can import pre-baked coursier caches, to run various Scala CLI commands.

We could add commands allowing to export (in a .tar.gz say) and import a whole coursier cache, either in the coursier CLI or in Scala CLI itself.

Users wanting to distribute such a cache would be expected to generate it with commands like

$ export COURSIER_CACHE="$(pwd)/cache"
$ scala-cli run … # run a few Scala CLI commands that you want to work fine with the distribution
$ scala-cli cache export -o cache.tar.gz 

To import the distribution, one would have to run things like

$ scala-cli cache import cache.tar.gz

An important point to pay attention to when implementing this would be not to blindly copy the files of a distribution in the local coursier cache, but rather group together files that are logically tied (if foo/a is in cache, then foo/.a.* is tied to it), and import each group as a whole (if foo/a is already there locally, compare foo/.a.checked to the one to be imported, and either don't import foo/a at all, or clear the local foo/a and linked files first).

[alexarchambault]

cc @lbialy

[lbialy]

This would be very useful for us right now.

[lbialy]

One question that I have is - would it take into account only dependencies required by any given project? I mean - exporting cache items that are required on per-project basis would be massively more useful. I have, for instance, several different projects for which I use different, often weird dependencies and there's no need to export all of my deps to customers, just the ones that project in question requires.

[lwronski]

One question that I have is - would it take into account only dependencies required by any given project? I mean - exporting cache items that are required on per-project basis would be massively more useful. I have, for instance, several different projects for which I use different, often weird dependencies and there's no need to export all of my deps to customers, just the ones that project in question requires.

Hi @lbialy,
yes, It will be possible to exporting cache items per project. I'am working now to prepare cache command to import and export coursier/bloop caches - #1123

[lwronski]

After analyzing the pros and cons of importing and exporting the cache, I have concluded that the risks outweigh the benefits.

While it is easy to implement, it raises security concerns. It would be necessary to add support for validating exported caches, such as checksum validation. Additionally, the main consideration is how this would be shared securely.

If there is access to the internet, it would be safer to download from the maven repository. If not, scala-cli should allow users to disable downloading anything with the --offline option, and use a local repository where users can verify the downloaded jars.

For this reason, I am closing this issue. However, if anyone still believes this mechanism is necessary, feel free to re-open the issue.