Transkun CLI — Piano → MIDI, running locally with Node.js + onnxruntime-node

The command-line (Node.js/TypeScript) version of the Transkun demo: feed it a WAV/MP3 file (or anything ffmpeg can read), and it runs the whole pipeline (mel front end → ONNX transformer → semi-CRF Viterbi decode) locally, with no browser involved, writing the resulting .mid file into the same directory as the source file.

Uses onnxruntime-node (native N-API binding, runs on CPU or GPU) instead of onnxruntime-web/WASM — noticeably faster than the browser demo on the same machine.

Install & run

npm install
npm run start -- ./song.mp3

(npm run start -- runs straight from source via tsx, no build step needed. On first npm install, ffmpeg-static and onnxruntime-node each download the binary that matches your platform.)

Or build once and run with plain node:

npm run build
node dist/index.js ./song.mp3

The MIDI file is written next to the source file, same base name: song.mp3 → song.mid.

Command-line options

transkun-cli <audio-file> [options]

  -o, --out <path>       Output .mid file path (default: same directory as the input file)
  -p, --provider <name>  Execution provider: cpu (default), cuda (Linux x64 + NVIDIA),
                          dml (Windows + DirectX 12, any GPU including AMD/Intel)
  --gpu                  Shorthand for --provider cuda
  --model-dir <dir>      Local cache directory for the ONNX model files (default: ./models/transkun)
  --model-base <url>     Where to download the model from (default: Hugging Face TuesdayCrowd/transkun-onnx)
  -h, --help             Show this help

On first run, the CLI automatically downloads roughly 60MB (transkun.onnx ~53MB, transkun-heads.onnx ~3.4MB, plus the small front-end buffers) into ./models/transkun/ and reuses it on subsequent runs — it won't re-download unless you delete the cache directory or point --model-dir elsewhere.

Running on GPU (CUDA / DirectML)

By default the CLI runs on CPU. Use the -p, --provider flag:

# Linux x64 + NVIDIA GPU:
npm run start -- ./song.mp3 --provider cuda
# (or the shorthand --gpu, equivalent to --provider cuda)

# Windows + any GPU (NVIDIA/AMD/Intel) via DirectX 12:
npm run start -- ./song.mp3 --provider dml

If the chosen provider fails to initialize (missing CUDA/cuDNN, driver mismatch, out of VRAM, ...), the CLI logs a warning and automatically falls back to CPU instead of crashing.

By OS (the prebuilt onnxruntime-node binaries support different GPU backends depending on the platform):

OS	GPU EP available via onnxruntime-node	Notes
Linux x64	cuda (and tensorrt, not wired up in this CLI)	Requires CUDA Toolkit 12.x + cuDNN 9.x installed system-wide. npm install automatically downloads the CUDA EP binaries (a few hundred MB) from NuGet — needs network access at install time. Skip this with npm install --onnxruntime-node-install=skip if you only plan to run CPU/have no NVIDIA GPU.
Windows	dml (DirectML)	cuda is not officially supported via onnxruntime-node on Windows; use dml instead — works on any DirectX 12–capable GPU (including a GTX 1050), no separate CUDA/cuDNN install needed.
macOS	cpu only	No official GPU EP via onnxruntime-node.

On VRAM: each 16-second segment produces a score matrix S of roughly ~172 MB and a context tensor ctx of roughly ~64 MB, plus the transformer's own intermediate activations — a 3–4 GB GPU (e.g. a GTX 1050) should still manage it, but it's fairly tight, especially if that GPU is also driving a display. If you hit an out-of-memory error on GPU, rerun with --provider cpu (CPU uses system RAM, and 32GB is normally plenty for this).

Stage logging

The CLI prints each step with a timestamp so you can follow progress:

[12:03:41.512] === Transkun CLI ===
[12:03:41.513] Input   : /home/user/song.mp3
[12:03:41.513] Output  : /home/user/song.mid
[12:03:41.514] Stage 1/4: checking for / downloading the ONNX model files ...
  [cache] params.json already present at ./models/transkun/params.json
  [download] transkun.onnx from https://huggingface.co/... ...
  [done] transkun.onnx — 53.10 MB in 8.2s
[12:03:50.021] Stage 1/4: creating an onnxruntime-node session (cpu) for transkun.onnx ...
[12:03:51.884] Stage 2/4: decoding audio via ffmpeg -> mono 44100 Hz ...
[12:03:52.310]   Audio is 214.3s long (9,450,630 samples).
[12:03:52.311] Stage 3/4: transcribing in 16s segments (8s overlap) ...
  [segment 0/27] extracting mel features (0.6s elapsed)
  [segment 0/27] running the transformer (ONNX) (1.4s elapsed)
  [segment 0/27] decoding semi-CRF (Viterbi) (1.6s elapsed)
  [segment 0/27] predicting velocity / sub-frame offsets (1.7s elapsed)
  ...
[12:05:02.114]   Done: 1842 notes, 96 sustain pedal presses.
[12:05:02.115] Stage 4/4: writing the MIDI file ...
[12:05:02.130] Done in 80.6s. Wrote: /home/user/song.mid

Source layout

• src/core/ — pure algorithm logic, shared with the browser demo (no dependency on Node or Web APIs): fft.ts, melFrontEnd.ts, viterbi.ts, midiWriter.ts, params.ts, buffersType.ts, melRanges.ts, transcriber.ts, modelInterfaces.ts (the IModelRunner/IHeadsRunner interfaces that keep transcriber.ts independent of onnxruntime-web vs onnxruntime-node).
• src/onnxModel.ts — implements IModelRunner/IHeadsRunner using onnxruntime-node, with GPU-provider selection and automatic fallback to CPU.
• src/audioLoad.ts — decodes audio via ffmpeg (bundled through ffmpeg-static, no need to install a system ffmpeg).
• src/modelCache.ts — downloads and caches the model files from Hugging Face.
• src/buffers.ts — loads params.json/freq2mels.f32/windows.f32/symbols.i32 from disk.
• src/index.ts — CLI entry point: argument parsing, stage logging, writing the MIDI file.

Limitations

• Runs on CPU by default (executionProviders: ['cpu']); pass --provider cuda/dml for GPU (see "Running on GPU" above). Falls back to CPU automatically if the chosen provider fails to initialize.
• Only the sustain pedal (CC64) is exported to MIDI, matching the browser demo and the reference implementation.
• Needs an ffmpeg binary matching your platform (Windows/macOS/Linux x64 & arm64) — ffmpeg-static downloads it during npm install; if that environment has no network access at install time, install a system ffmpeg instead and adapt src/audioLoad.ts to call the one on your PATH.