μJSuite is a tool that can run different workloads on different IoT-friendly JavaScript engines. Each engine is automatically built from source and gets executed inside a Docker container. This tool can be used to generate benchmarks and compare the performances of different engines.
You can find collections of generated benchmarks in this repository.
To run μJSuite, all you need is Docker to be installed on the target machine.
Make sure the perf_event_paranoid system variable is set to 2 or lower, as some Linux distributions seem to set it higher by default. This is required for μJSuite to be able to measure performance statistics. To fix this, run the following command:
sudo sysctl -w kernel.perf_event_paranoid=2
There's no need to manually setup μJSuite. The first time you run the command line tool, it will automatically download dependencies and build the source code.
Clone or download this repository, then run bin/mjsuite (or bin\mjsuite on Windows) from the root of the project. To get a list of available commands and options, run bin/mjsuite help.
When requested as an argument or option, a workload must match a script in the workloads directory (without the .js extension), and an engine must match one of the folders in the engines directory.
All available commands are described below. Each of them provides a --help option that displays additional information about their usage.
bin/mjsuite benchmark [options]
bin/mjsuite bm [options]
This command runs all workloads with all engines and generates a benchmark. The results are then written into a JSON file. You can also specify which workloads and engines to run using the options below.
When encountering a new engine, μJSuite will automatically download its source code and build a Docker image. This will take some time to complete, depending on the engine.
The following options are available:
-w,--workload <workloads...>: the workload(s) to run (default: all).-e,--engine <engines...>: the engine(s) to use (default: all).-o,--output <filename>: the output file that will store the results.-p,--plot: display plots immediately after the benchmark is generated.
The --workload and --engine options also support negative filtering. For example, running bin/mjsuite benchmark --engine !jerryscript will select every engine except JerryScript.
To list all available engines, run:
bin/mjsuite engine list
To download and build a specific engine, run:
bin/mjsuite engine setup <engine>
Note that this is done automatically when the benchmark command encounters a new engine. Running this command again will rebuild the engine and overwrite the existing image.
To list all available workloads, run:
bin/mjsuite workload list
To draw plots for an already generated benchmark, run:
bin/mjsuite plot <benchmark>
where <benchmark> is a path to the target JSON file (previously generated by the benchmark command).
❯ bin/mjsuite benchmark --workload array-sort --engine jerryscript --plot
Running workload 'array-sort' with engine JerryScript
Workload finished in 853 ms
Saved results to benchmark_2023-06-12_19-29-22.json
Running bin/mjsuite will start μJSuite inside a Docker container, which may cause errors on ARM machines. If you get errors during the setup about lzma-native, try installing Node.js manually, then install dependencies, build the source code and run μJSuite by calling node directly:
npm install
npm run build
node build/main <arg...>
Drawing plots is also not supported inside Docker containers, so follow the same process if needed.
To add a new engine to the project, create a new folder in the engines directory. This folder must contain a manifest file, a Dockerfile and a workload template (optional).
Create a manifest.json containing an object with the following fields:
name: the name of the engine.repository: the GitHub repository of the engine, in the formuser/repo.version: a git tag referencing the target version. If theshaproperty is provided, this becomes just informative.sha(optional): the target commit SHA hash, which will overwrite the version tag. Helpful when the repository doesn't provide tags.source(optional): a URL to the source code. If set, μJSuite will download from this URL instead of GitHub. Helpful with embeddable engines that provide pre-processed packaged source code.clone(optional): set totrueto clone the repository instead of simply downloading it. This will take more time, but some engines require the source code to be in a git repository in order to be built.
Note: when running μJSuite, the engine argument must match the name of the folder, not the name specified in the manifest.
The Dockerfile must describe how to build the source code of the engine. When executed, the Dockerfile receives srcPath as an argument, which contains the full path to the source code. This argument can be used to copy the source code into the image (e.g., COPY $srcPath ./)
Once the engine is compiled, we want the image to only contain the executable (and its dependencies, if any) without all the build tools and cache. This can be achieved with multi-stage builds. We also need the final image to include both the Perf and GNU Time tools. Make sure to provide a version of GNU Time that is recent enough, as older versions are known to report wrong measurements.
The entry point of the container must be set to the engine executable file and must not include arguments.
Since different engines can have different specifications, some of them may need some adjustments before being able to run a workload (e.g., the console.log function could be named differently). If such adjustments are needed, we can create a template.js file with the following syntax:
// code being executed before the workload
const console = { log: print }
// workload will be inserted here
${workload}
// code being executed after the workload
print(duration)If no adjustments are needed, there's no need to create this file.
To add a new workload, simply create a JavaScript file in the workloads directory. As a convention, workloads should start with the line var N = #, which specifies the number of iterations in the workload.
Workloads should follow older ECMAScript standards so that most engines will be able to run it. Note that engines that can't process a workload will be skipped during benchmark generation.
For now, workloads cannot depend on external source files or assets (i.e. images). The whole code and data must reside inside a single JavaScript file.
If μJSuite fails to work as expected, you can run it with the --verbose option. This will increase the amount of information being printed to the console, which can be helpful to resolve problems.