Plugin system RFC

[manio143]

Plugin system RFC

A long await solution to #13. The goal of this document is to provide a description of a plugin system, both in terms of usability and user experience and in terms of steps necessary for its implementation. The discussion underneath should result in materializing a concrete task list which will be implemented over time.

TL;DR

Let's include a Plugins property to Package definition that defines additional assemblies loaded by GameStudio and AssetCompiler. And a bunch of other things to help write/use plugins. It sounds simple but there's a lot of work behind the scenes to make it happen.

Motivation

Stride offers a lot out of the box, however, there's many instances where a user may want to execute custom actions in the GameStudio to help their tasks, have custom assets and custom editors, without necessarily having to recompile Stride from sources. Moreover, allowing plugins would enable Stride users to share their custom code without having to extend the engine in the main repo, or doing painful merges of forks. Also, a plugin oriented architecture should allow for easier development down the line.

Extensibility points

It's important to highlight areas that extensibility will focus around.

Runtime

Runtime libraries, i.e. libraries with components, processors and systems are already in a decent state when it comes to extensibility. Previous work has been done to enable e.g. Stride.Physics, Stride.Video to depend on Stride.Engine and be optional for the user to include. Following their pattern users can create new runtime libraries, inclusion of which will allow their code to be executed. Therefore, this document will not look into further extensibility points to the runtime.

Asset pipeline

The assets related projects (assets compiler and its dependencies) have been designed in a modular way that allows for creating custom assets and custom assets compilers. I've previously shown to be able to create custom assets (here), however, it required my game to take a dependency on Stride.Core.Assets, which currently is compiled under StrideEditorTargetFramework (windows only). The later outlined idea of design projects allows to skip fiddling with the build system and greater cross-platform compatibility.

Editor

The Stride GameStudio is the central point for any asset related operations. It has the most potential for extensibility and is also the least familiar to the users in terms of its internals. The following are some things we would want to enable in plugins:

• Custom entity factories (derived from EntityFactory)
• Custom asset factories (derived from AssetTemplateGenerator)
• Custom asset editors (derived from AssetEditorViewModel and annotated with [AssetEditorViewModel]) with all the parts (view, asset viewmodel, asset node updater)
• Custom asset preview (derived from PreviewFromEntity or PreviewFromSpriteBatch and annotated with [AssetPreview])
• Custom asset thumbnail (derived from ThumbnailCompilerBase and annotated with [AssetCompiler] under ThumbnailCompilationContext)
• Custom component gizmo (derived from EntityGizmo)
• Custom in editor action (e.g. a custom menu)

The good news is: there already is an extensibility point for most of those things - StrideAssetsPlugin allows to explicitly register anything that isn't being picked up by attribute or as an interface implementation. A custom plugin just needs a good entry point to be registered (see test for a less than ideal solution).

Usability scenarios

When talking about a plugin system we need to consider how users will interact with it. The following scenarios describe the proposed solution of referencing plugin assemblies.

• External plugin - the user includes a reference to a NuGet package in their game project. This package will contain runtime classes. Another package would contain the plugins to allow the user to work with assets later used in their game. A potential way to include the design package would be to extend Package (sdpkg file) with a Plugins property that is a list of {Name: version} NuGet packages resolved and loaded as plugins.

• Internal plugin - the user creates a project in their solution that references their game project and Stride.Assets.Presentation (or just Stride.Core.Assets), but that project is not referenced by executable projects. The project is also added to the Plugins list in the game's sdpkg file. This project is considered for assembly reloading and any assets defined in it must be fully reloaded (or not - this could be a flag).

Considerations

• Editor plugins are currently only additive - internal user plugins with reloading would require modifications to allow plugin removal and dynamic reinitialization - for the stage 1 of implementing plugins I would disable reloading even for local plugins, but this has an unfortunate implication that none of the projects a plugin depends on can be reloaded either or things will break.
• We can allow for loading any NuGet package into memory as a plugin, but should we have them register themselves in the AssemblyRegistry under Assets category, or would this be done by the plugin loader? Ideally we would want to block Module initializer execution for safety reasons, but that has been used a lot in Stride own libraries. Now that I think about it, safety goes out the window any time we execute any user code, but at least it would have to be implemented with Stride.
• Error handling when plugin is unavailable to be resolved
• Should there be an attribute for editor plugins discovery?
• We should add an option in GameStudio to generate a plugin project and add reference to it into the game project.
• Good documentation will be crucial for success of plugins.
• Searching for plugins - maybe a "strideplugin" tag on NuGet.org? We could include a plugin browser in the editor for users to add them directly in the editor. There would have to be something that link the runtime package with the design package.

Task list

To be added after a discussion.

[ykafia]

I wonder if this work would still be useful if we decide to switch editor in the future?

[manio143]

I wonder if this work would still be useful if we decide to switch editor in the future?

Yes, to an extent. There are changes around the package management system which are shared between the asset compiler and the editor - this is the entry point for loading plugins. If we do end up rewriting the editor, it will be up to the owner of a plugin to migrate their implementation onto the new version of presentation classes. But I suspect a good chunk of plugins will be only focused around {runtime component, asset, asset compiler} as the editor part is the one that takes a bit more time.
Also if people do start to author plugins it will be easier to migrate onto new UI (and for the designers of the new editor to see which features are desired by plugin authors), then to wait with enabling plugins until after the potential switch.

[Eideren]

@manio143 I do think that a very large part of the use-case, if not most, will fall into level-design (augmenting the scene view, procedural scene composition, tools for blocking out levels, etc.) so working on passing editor input to the user, having a way to draw things on screen and in tool windows and interacting with the live scene view. Those things are intrinsically tied to the UI system.
I might be wrong though so if some of you guys could chime in with what you would do with this plugin system that would help.

Or we could use a different UI system for plugins specifically which we would migrate the rest of the editor to later down the line ?

[manio143]

For me the number one thing is the custom data assets - as you remember I worked on integrating something into Stride, but I had to do a lot not so nice changes to the internals of data serializers to handle generic classes and got hung up on the asset reloading on assembly reload - this is something that can be leveraged with the plugin system and roslyn code generators.

But yes, I have to agree that a large number of highly reusable plugins would attempt to make certain in-editor tasks easier, thus getting tied to the UI. I don't exactly see how we could use a different UI system for just plugins. It depends on what the plugin author wants to achieve - custom property editors have to be written in XAML as they would be rendered in the property grid, but advanced gizmo's could even use Stride's in-game UI system for modifying components.

[Eideren]

custom property editors have to be written in XAML as they would be rendered in the property grid

We could have a dock-able empty wpf element that another ui system renders to, I haven't investigated this yet but I'm sure it's feasible with more lightweight library like myra or imgui. Once/if we swap the editor ui we would have to swap the surface with whatever UI we end up using and it wouldn't break user UIs.

There is also the issue of giving users a way to change the scene, right now the only way to do so in engine is painful to use and extremely slow because of the multiple scenes synchronizing and serialization round-tripping the engine does on every changes, we might have to simplify that or gut it out before moving forward with scenes plugins.

Should there be an attribute for editor plugins discovery?

How the engine figures out which assembly is a plugin ? So we would ask of users to specify something like this in their plugin's csproj <AssemblyAttribute Include="Stride.Core.PluginAssembly"/> and then test that attribute in engine or something to that effect, I'm sure xen2 has some insight on this.

[manio143]

It's not a bad idea, with the rendering part to be handled by another lib. I will have to think about how exactly the interface layer for the plugin creator would look like so that it's not too much pain to execute view model updates from the other UI lib.

Should there be an attribute for editor plugins discovery?

This was targetted at how a custom StrideAssetsPlugin instance would be registered (should've worded it better) - in general we should try to make all things registered with attributes as this makes it often easier to discover in one place, rather than have the user call an explicit Add or Register method in their code.

The point you mentioned - we can have a custom [assembly: Stride.Core.Plugin] attribute and read that off of a loaded assembly.

[xen2]

Thanks for the well thought summary/proposal!

You identified one of the main design decisions to take: how to express what is a design-time plugin and make it fit with NuGet with proper dependencies?

As you mentioned, one possibility that sounds quite straightforward for the user to understand is to have 2 user projects: 1 runtime project and 1 design project.

If we take this approach, there is likely an explicit dependency (ProjectReference): the design project depends on the runtime project (i.e. the runtime project defines the runtime class of a custom asset).
However, there is also an implicit dependency: the runtime project depends on the design project for asset compilation.

Runtime project is compiled in two phases: code compile, then asset compile.
We need somehow to teach the build system to compile this design project after runtime code is compiled (runtime DLL might be needed for ProjectReference) and before assets can be compiled (design DLL needed by asset compiler).
Basically, it would go in this order when compiling a runtime project:

1. Compile Code (generate DLL, contains user custom runtime classes)
2. New: Build design projects needed by asset compiler which depends on DLL from step 1 (user custom assets, etc.)
3. Compile Assets (using both libraries)

It can be done by either MSBuild or by our asset compiler (it will need some careful considerations to figure out which is best).

Of course, we could think of another way to structure the projects and how they depend on each other (i.e. introduce an additional project for shared code).
Let me know if you have any ideas on that front.

[ykafia]

One aspect of the engine that needs plug-in support would be the post process fx

[Aggror]

Hi @manio143,
Would you be prepared to summarize this awesome suggestion for the Epic grant proposal? personally I think this topic is in the top 3 of most needed features of Stride. It would be great if we had budget for someone to start addressing this.

[manio143]

I've written a short high level explanation https://hackmd.io/@Ed53RzBbTu2HT46rCM--rQ/Bk0ukH_Ct

[Aggror]

wow that is really well formulated.