Allow loading external Rust Filters dynamically

[markmandel]

Context:

If the filter traits were moved to another crate, filters could be defined in separate crates and loaded by Quilkin at runtime. That would be much more reasonable for users.

Originally posted by @XAMPPRocky in #401 (comment)

This is an issue to track this idea, as I think it's definitely got value for sure. I'm sure @XAMPPRocky you know more about this than I do. I couldn't find a particularly clear guide.

Here are the resources I found:

• https://adventures.michaelfbryan.com/posts/plugins-in-rust/
• https://michael-f-bryan.github.io/rust-ffi-guide/dynamic_loading.html
• https://zicklag.github.io/rust-tutorials/rust-plugins.html (deprecated)
• https://github.com/rodrimati1992/abi_stable_crates (looks like this is the way?)

[XAMPPRocky]

I hadn't looked at stable_abi closely before, it seems interesting and worth investigating. I think we should discuss what kind of behaviour we expect from the runtime loading. Here are a couple of open questions that come to mind.

• Where do we store the plugins? What's considered best practice for dynamic libraries in Linux/macOS/Windows?
  • I know Windows autoloads dlls adjacent to the running binary.
• When are are plugins loaded, at startup, dynamically?
  • This is important as it would affect how the FFI code would work. stable_abi doesn't support unloading for example.
  • I'm inclined towards startup only for simplicity and security.
• What context do we send across the FFI boundary?

[markmandel]

Where do we store the plugins?

My suggestion: make the folder to search (I'm assuming there may be more than one plugin loaded at once) configurable through the config yaml, but with a default of a separate relative directory from the main binary.

I'm inclined towards startup only for simplicity and security.

100% concur. Runtime replacement of code scares me. 😱

What context do we send across the FFI boundary?

I expect this will require experimenting with abi_stable_crates and seeing what is possible, and how close it might come to our existing context data.

Side thought: Assuming all goes to plan with #411 -- would that make things harder to load an external plugin, as we no longer have a context object (or would we need to fabricate one just to allow external loading of code?) Again, might just require some experimentation to see what is possible.

[markmandel]

https://robert.kra.hn/posts/hot-reloading-rust/

Was just reading this - based on work Bevy has been doing