Plugins
This wiki has been updated for v1.4.3. An update for v2.x will come very soon. Contact me if you see this meanwhile and have any questions. (most details are the same, v2 just has some additional new events and features)
This is a quick developer reference for how plugins work in xVASynth. Document structure:
- Brief intro/overview
- Enabling plugins
- The plugin.json file
- Back-end Python plugins
- Front-end JavaScript plugins
- Event Reference
- Custom Events
The most important thing to remember is that the xVASynth app is actually split into two programs. A front-end Electron app for the UI, running JavaScript (Nodejs), and the backend program written in Python, which takes care of the models inference and ffmpeg use. The two halves of the codebase communicate via a local HTTP server.
Plugins for xVASynth can be written for either (or both) of these halves. Anything that changes the front-end (what the user sees) will need to be written in JavaScript file(s), and anything for the backend needs to be written in Python file(s). The app supports only Python for the back-end at the moment, but other languages are also fine, so long as you can exec() their files from within a small python script.
The front-end also supports additional .css file(s).
For whichever part of the program you wish to add a plugin to (front/back-end), the app will register your plugins, and will execute specific functions from the source files, at specific "event points". You just need to specify which function in which files should be executed in what event (in the plugin.json file, more on this later). You can also specify a setup function, which will only run once, whenever the plugin is first registered by the app.
You can also specify a custom event, by adding hooks to, and calling a "custom-event" event (detailed below).
Plugins are installed in the <.exe location>/resources/app/plugins/ folder. Try to come up with a short id for your plugin, which you don't think will easily conflict with plugins other people will make. Name the ID using alphanumerical and underscore characters, without spaces, to make things easy.
Plugins detected by the app are listed in the plugins menu (opened from the puzzle piece icon in the top right of the app).
Here, they can be enabled/disabled by selecting/deselecting the checkbox on the first column. The remaining columns display metadata from the
plugin's plugin.json file. By first selecting a plugin (by clicking on a line), the order can be adjusted using the Move Up and the Move Down buttons, and confirmed by clicking the Apply button.
Plugins do not overwrite each other, depending on order. The order dictates the order in which plugins' function(s) get called, when an event is called, for which multiple plugins have function hooks. This likely won't be important, but will be needed for example when a plugin may depend on the output of a different plugin. CSS style files DO effectively overwrite, however, due to how browsers work.
This file is the manifest where your plugin's actions are defined, and bound to whatever events you wish to add functionality to. The following is a minimal example for registering a plugin (though there is no functionality included):
{
"plugin-name": "Test plugin",
"author": "DanRuta",
"nexus-link": null,
"plugin-version": "1.0",
"plugin-short-description": "A test plugin to develop the functionality",
"min-app-version": "1.0.0",
"max-app-version": null,
"install-requires-restart": false,
"uninstall-requires-restart": false,
"front-end-style-files": [],
"front-end-hooks": {},
"back-end-hooks": {}
}
The available keys are detailed in the table below:
| Key | Purpose | Is required | Data type | Example |
|---|---|---|---|---|
| plugin-name | This is your plugin name. Make it short and sweet. | Yes | String | "Test plugin" |
| author | Display credits | No, but recommended | String | "DanRuta" |
| nexus-link | Compatibility with future plans for the app | No, but STRONGLY recommended, if you do publish it on the nexus | String | "https://www.nexusmods.com/skyrimspecialedition/mods/44184" |
| plugin-version | This is a semantic versioning field to keep track of your plugin's version. This does not get used by the app (except for displaying in the plugins panel), and is just for your book-keeping. | No, but STRONGLY recommended | String | "1.0" |
| plugin-short-description | This is a SHORT description of your plugin, for making it easier for the user to know which plugin they are enabling/disabling in the panel. | No, but STRONGLY recommended | String | "A test plugin" |
| min-app-version | Semantic versioning for a minimum required APP version. If a user is running an app version lower than this, the plugin cannot be enabled | No | String/null | "1.0.0" |
| max-app-version | Similar to min-app-version. Plugin cannot be enabled if the app version is larger than this. | No | String/null | "1.1.0" |
| install-requires-restart | Toggle to prompt users to re-start the app after enabling the plugin in the plugin menu, to enable functionality. | No | Boolean | false |
| uninstall-requires-restart | Toggle to prompt users to re-start the app after disabling the plugin. | No | Boolean | false |
| front-end-style-files | A list of .css files to append to the bottom of the app's style sheet. | No | Array of strings | ["style1.css", "style2.css"] |
| front-end-hooks | Where you define function hooks for the front-end. See the Front End section later down. | No | Object | See section |
| back-end-hooks | Where you define function hooks for the back-end. See the Front End section later down. | No | Object | See section |
To assign files and functions to events in the back-end-hooks object, you need to follow this format:
"back-end-hooks" : {
"<event_name>": {
"pre": {
"file": "yourFile1.py",
"function": "do_something"
},
"post": {
"file": "yourFile1.py",
"function": "do_something_else"
}
},
"<other_event>": ...
...
}
Each event has the same data structure, with a pre and post event. You can pick to write just one of them, or both. The pre event is kicked off before the reference event, and the post is kicked off afterwards. The only exception to this are custom events (see later down).
As an example, for the output-audio event, you could include a pre event to carry out some changes to the temporary audio file first, before ffmpeg is used to apply the user's audio settings to the final audio file. You could then use a post event to run some further script AFTER ffmpeg has output the final audio (eg generating a .lip file).
You can use however many python files you wish, with as many functions referenced in them as you wish. But the files have to be python files (you can exec() other files from python, if you wish to use other languages). You can name your files whatever you wish.
Boilerplate code for integration is minimal. All you need to do in your python file is write your functions. The function names just have to match what you have specified in the plugins.json file. The following is a simple example of what a python file could look like:
# OPTIONAL
# ========
logger = setupData["logger"]
isDev = setupData["isDev"]
isCPUonly = setupData["isCPUonly"]
appVersion = setupData["appVersion"]
# ========
def do_something(data=None):
print("before the event")
def do_something_else(data=None):
print("after the event")
The do_something and do_something_else functions are what is referenced in that plugin.json example. The scripts, when initialised, have access to a global setupData object, containing data from the program. The available keys are logger, appVersion, isDev, and isCPUonly. The isDev boolean indicates whether the app is running in development mode (true if running the GitHub code, false if using a compiled build).
If you need to execute some code once, to initialise the plugin, but NOT whenever the plugins are refreshed (happens every time the Apply button is clicked in the app, following any change to the plugins list), you can optionally write a function named setup. The setup function accepts a data object parameter, containing the same logger, appVersion, isDev, and isCPUonly keys as above. You do not need to include this function in plugin.json, but it is important that this function is named setup. As example:
# OPTIONAL
# ========
def setup(data=None):
logger.log(f'Setting up plugin. App version: {data["appVersion"]} | CPU only: {data["isCPUonly"]} | Development mode: {data["isDev"]}')
# ========
If you need to run a function when the app is being disabled, you can write a teardown function, similar to the setup function. This again does not need to be specified in the plugin.json file, but you must name the function teardown. This should be useful for any cleaning up you may want to do, or if you need to revert any changes to the app you have made, if any. This function receives the same parameters as the setup function: logger, appVersion, isDev, and isCPUonly
# OPTIONAL
# ========
def teardown(data=None):
logger.log(f'Uninstalling plugin. Cleaning up. {data["appVersion"]} | CPU only: {data["isCPUonly"]} | Development mode: {data["isDev"]}')
# ========
Plugins have access to the server.log logger. You should use this for debugging, if developing around a compiled version of xVASynth, rather than using the development files from the GitHub repo. All your messages are prepended by "[<your_plugin_id>]", to clarify to everyone where each message is coming from. Avoid spamming the log files, as that is detrimental to the user, if they need to debug anything using the log file.
To log a message, you use logger.log(message), with a single string parameter for the message to log. You can see the above example.
Module imports work as normal in the python files. However, there is a slight catch. To make sure the scoping works ok, you need to access imports globally, in your functions. For example, this is what you need to write, to use the os module in a function:
import os
logger = setupData["logger"]
def a_function(data=None):
global os, logger
logger.log(", ".join(os.listdir("./")))
To import a custom module (either your own files, or a library which you must place in your plugin's folder), you need to prepend the import path as follows (the example is for a plugin with the plugin id = test_plugin, and a secondary file called second_file.py, in which there is a function called doImportableFunction):
from plugins.test_plugin.second_file import doImportableFunction
def a_function(data=None):
global doImportableFunction
doImportableFunction()
Important:
In compiled releases of the app, the app files have "resources/app" prepended to their path. In order for these imports to work in the release, you need the following code added to your script before the imports:
isDev = setupData["isDev"]
if not isDev:
import sys
sys.path.append("./resources/app")
Similar to the back-end-hooks object, the front-end-hooks object contains a number of events, with a pre and/or post sub-event, each of these containing the function name and its file. The events will be different, but everything else is the same, other than having to use JavaScript files, instead of Python (you can still include however many files you wish, with whatever name(s), with however many functions in each file as you wish).
Similar to the python code, the boilerplate is minimal. To expose functions to xVASynth, you need to add them to the exports object. Here is a simple example:
"use strict"
const preKeepSample = (window, data) => {
console.log("preKeepSample data", data)
}
const postKeepSample = (window, data) => {
console.log("postKeepSample", data)
}
exports.preKeepSample = preKeepSample
exports.postKeepSample = postKeepSample
Similar to back-end Python plugins, you can optionally include a setup function for front-end plugins, to define a one-time-per-app-session function. Like the python setup function, this will run only once, when the app starts, or when the plugin is first enabled, but not again, until the next time the app is started. The function again needs to be called setup, and does not need to be explicitly defined in plugin.json. Example:
// Optional
// =======
const setup = () => {
window.appLogger.log(`Setting up plugin. App version: ${window.appVersion}`)
console.log("setup")
}
// =======
exports.setup = setup
There is an optional teardown function in front-end plugin code, similar to the back-end code. As before, this optional function runs once, when the plugin is uninstalled, and should be used for any cleaning up. It is especially useful on the front-end, as you may need to use it to undo any UI changes made in the plugin. This function does not need to be defined in the plugin.json file, but must be named teardown.
// Optional
// =======
const teardown = () => {
window.appLogger.log(`Uninstalling plugin. App version: ${window.appVersion}`)
console.log("teardown")
}
// =======
exports.teardown = teardown
Front-end plugins have access to the app.log logger. Like with the server.log logger for the back-end plugins, all messages are prepended with your plugin ID, to keep track of where messages are coming from. As the window object is globally available, the logger instance can easily be accessed as window.appLogger. You can also use the normal console.log if you are using the development code of xVASynth. Again, use the logs sparingly when deploying, to avoid log clutter.
Any .css you want to add to the front end can be written in any number of .css files. You can include these in package.json under the front-end-style-files key (relative file path). These are appended at the bottom of the app's style sheet, so they should overwrite.
This is a reference for the back-end and front-end events. The list will likely grow over time, if/when more events are added in.
| Event Name | App version | Pre/Post | Description | Data |
|---|---|---|---|---|
| start | 1.4.0+ | Pre | This is kicked off as soon as the app is starting up, before user settings, FastPitch, vocoders, or the local HTTP server are initialised | None |
| start | 1.4.0+ | Post | This is kicked off after the app backend has finished starting up | None |
| load-model | 1.4.0+ | Pre | Before a voice model is loaded | String: the full path to the checkpoint file |
| load-model | 1.4.0+ | Post | After a voice model is loaded | String: the full path to the checkpoint file |
| synth-line | 1.4.0+ | Pre | Before a line is synthesized | {"sequence": String, "pitch": [float], "duration": [float], "pace": int, "outfile": String, "vocoder": String} |
| synth-line | 1.4.0+ | Post | After a line is synthesized | {"sequence": String, "pitch": [float], "duration": [float], "pace": int, "outfile": String, "vocoder": String} |
| output-audio | 1.4.0+ | Pre | When the Keep Sample button is clicked, if using ffmpeg. Before the final audio has been output via ffmpeg. |
{"input_path": String, "output_path": String, "game": String, "voiceId": String, "voiceName": String, "inputSequence": String, "letters": [String], "pitch": [float], "durations": [float], "vocoder": String, "audio_options": {"hz": String, "padStart": int, "padEnd": int, "bit_depth": String, "amplitude": float}} |
| output-audio | 1.4.0+ | Pre | When the Keep Sample button is clicked, if using ffmpeg. AFTER the final audio has been output via ffmpeg. |
{"input_path": String, "output_path": String, "game": String, "voiceId": String, "voiceName": String, "inputSequence": String, "letters": [String], "pitch": [float], "durations": [float], "vocoder": String, "audio_options": {"hz": String, "padStart": int, "padEnd": int, "bit_depth": String, "amplitude": float}} |
| batch-synth-line | 1.4.1+ | Pre | Before a line is synthesized via batch mode | {"speaker_i": int, "vocoder": String, "linesBatch": [ [ Strings ] ], "batchSize": int, "defaultOutFolder": String} |
| batch-synth-line | 1.4.1+ | Post | After a line is synthesized via batch mode | {"speaker_i": int, "vocoder": String, "linesBatch": [ [ Strings ] ], "batchSize": int, "defaultOutFolder": String, "req_response": String} |
[output-audio] The "pitch" and "durations" arrays are empty on the first user generation, as the editor is empty at that point.
| Event Name | App version | Pre/Post | Description | Data |
|---|---|---|---|---|
| start | 1.4.0+ | Pre | This is kicked off as soon as the app is starting up, before anything happens. | None |
| start | 1.4.0+ | Post | This is kicked off as soon as the app is done starting up, after the FastPitch/WaveGlow/server loading modals are gone. | None |
| keep-sample | 1.4.0+ | Pre | When the user clicks the Keep Sample button, regardless of ffmpeg use. |
{"from": String, "to": String, "game": String, "voiceId": String, "voiceName": String, "inputSequence": String, "letters": [String], "pitch": [float], "durations": [float], "vocoder": String, "audio_options": {"hz": String, "padStart": int, "padEnd": int, "bit_depth": String, "amplitude": float}} |
| keep-sample | 1.4.0+ | Post | When the user clicks the Keep Sample button, regardless of ffmpeg use. After the audio has been output, unless there was an error. |
{"from": String, "to": String, "game": String, "voiceId": String, "voiceName": String, "inputSequence": String, "letters": [String], "pitch": [float], "durations": [float], "vocoder": String, "audio_options": {"hz": String, "padStart": int, "padEnd": int, "bit_depth": String, "amplitude": float}} |
You can additionally call a custom python event from a front-end plugin. For example, in the front-end plugin, you may add a button which you'd like to link to some python code. Instead of hooking a function on a pre-defined event, you can call a separate event, which will only be active for your own plugin (other plugins can also add python hooks for custom events, but they will not be called).
Check the example plugin for a complete set-up for what custom events look like. Briefly however, you need to use the following in your front-end code to call the custom event, and you MUST add your plugin id to the data body:
fetch(`http://localhost:8008/customEvent`, {
method: "Post",
body: JSON.stringify({
pluginId: "plugin_id", // The plugin id here is required
data1: "some data",
data2: "some more data",
// ....
})
})
Your plugin.json file needs to contain the following:
"back-end-hooks": {
"custom-event": {
"file": "main.py",
"function": "custom_event_fn"
}
}