Plugins should be able to depend on third party libraries

[hlxid]

Description

Currently plugins can't have dependencies on other libraries. When added to their build.sbt file they compile fine and IntelliJ auto-completion works. But when they get run and try to use a class from their dependencies they throw a java.lang.NoClassDefFoundError. It's not working because we use the package sbt task which only adds the class files generated from the local sources to the jar file, but doesn't include the class files from the dependencies.

According to the wiki plugins should be able to have dependencies:
https://github.com/codeoverflow-org/chatoverflow-wiki/blob/dd7688339a33fd0f114512e63f4d6d00255bc94e/development/Writing-a-plugin.md (end of the first paragraph of the "Create a new plugin project" section)

Some ideas for a solution

A solution to this is to use the assembly task that is added by sbt/sbt-assembly. However the assembly task currently fails while packaging the framework because some files are provided by 2 or more libraries. We either need to get the assembly task on the framework working or we need a custom task that only executes assembly in the plugin sbt projects, but I currently don't know how such thing could be done. I think that it is doable though.

This solution has some disadvantages. First if each plugin has a copy of the library in its jar and 10 plugins use the same dependency we have a really inefficient use of storage.
The second problem is that I don't know if the JVM is smart enough to check that those 10 copies of the library are the same and only keep one copy in memory or if it will then keep all 10 version of the library in memory. This might really become a problem with a huge count of running plugins on a low-memory-environment like a Rasperry Pi 3 which only has 1 GB of RAM and already looses at least 200-300 MB to the operating system and JVM. In my testing I had arround 430 MB of used memory with no running plugins or connectors, but I also had containerd and a k3s node with some pods running meaning I wouldn't take this number serious at all.

Another idea I had is to add a xml file containing all dependencies into the plugin jar and then download them at runtime using lets say coursier. This would allow for sharing the downloaded dependencies between plugins which drastically decreases storage needs, but I think this is a really big overkill for what we are trying to achieve.

I'm open for any thoughts/ideas on this.

[J0B10]

I'm already familiar with this problem from developing plugins for spigot minecraft servers.

There if two plugins used the same library it lead to errors.
A common way to fix it was using a maven plugin to reshade the class paths (e.g. net.dv8tion.JDA -> org.codeoverflow.testallplugin.lib.dv8tion.JDA)
wich still had the disadvantage of wasting RAM.

I don't know how different class loaders handle this, maybe there are some that fix the issue.
I'm pretty sure a custom class loader could be used to check when loading the same class twice.

I really like your second idea!

[sebinside]

When we first designed the multi-sbt-project build and runtime loading of plugins, we already had the idea of plugin dependencies. In this time, we tested registering custom dependencies and everything worked as intended. But that was... 2017? 2018?

The framework already uses the assembly task, but only for the bootstrap launcher, because this one needs the scala library embeded. for every plugin this seems overkill - and like you both said, a potentially huge waste of memory.

We should probably find a solution based on the plugins build.sbt file (pluginName.sbt), because we would get IDEA support for free. But aside from the assembly task... I really don't know anything that is scalable enough. Writng custom logic to retrieve dependencies (and resolve them properly incl. version checking) based on a XML file seems like even more overkill to me 😅

[sebinside]

That beeing said, coursier could be a solution for this. But then, the dependencies should probably be retrieved from the sbt file, not a custom file.

[hlxid]

My idea was to create a sbt auto-plugin that would generate a maven compatible pom (using makePom, a integerated sbt task) and add it to the jar when package is run on a plugin project.

Coursier has a integrated pom parser that is pretty easy to use. From there we can just use the coursier api to fetch all dependencies to the local cache and get a list of jars that need to be included in the class loader.