Edit video by editing the transcript. Drop a clip, Whisper transcribes it locally in your browser, click words to delete them, and the preview plays the cuts in real time through a WebGL2 shader pipeline plus a face-tracked reframe / captions / overlays stack.
pnpm install
pnpm devOpen http://localhost:3000.
For the live teleprompter and Tonos draft features, copy .env.example to .env.local and add a DEEPGRAM_API_KEY. Everything else works offline.
- Landing page → Upload your own or Try with sample
- Drop a talking-head clip (mp4/webm/mov). Whisper-small.en loads the first time (~150MB, cached) and transcribes locally — audio never leaves the browser
- Click words to delete; gap chips show silence / "uh"s detected from audio energy
- Hit play — the preview skips deleted ranges with frame-accurate seek and crossfaded audio
- Add captions, face-tracked reframe, shader looks, or B-roll overlays from the toolbar
- Re-record any segment in-browser; the new take splices into the timeline as a separate clip
- Export bakes the canvas + faded audio into mp4 via
MediaRecorder
| Piece | Tech |
|---|---|
| Local transcription | @huggingface/transformers Whisper-small.en in a Web Worker, audio decoded via OfflineAudioContext, resampled to 16 kHz mono |
| Cut / splice model | VideoClip[] per track with (sourceIn, sourceOut) + (projectStart, projectEnd); removeSourceRange / insertSourceRange / splitClipAt operate on metadata only — source files are never re-encoded |
| Skip-playback | requestVideoFrameCallback loop seeks past deleted ranges, hops between clips, runs a wall-clock gap clock for empty timeline regions |
| Multi-track timeline | Talking-head + demo tracks, drag-to-reorder clips, split-at-playhead, per-clip fade ramps |
| Face tracking | MediaPipe tasks-vision landmarker → smoothed reframe transform on the compose canvas |
| Background segmenter | MediaPipe selfie segmentation for the bg-blur shader preset |
| Shader preview | Hand-rolled WebGL2 (no three.js): fullscreen-triangle, <video> uploaded as texture each frame, six presets switched via uPreset uniform |
| Captions / overlays | Canvas-rendered, time-indexed, animated via project-time keyframes |
| Live teleprompter | Deepgram streaming WebSocket; server route mints short-lived tokens so the long-lived key stays server-side |
| Tonos draft / rewrite | OAuth (PKCE) into tonos.fyi for voice-profile-aware text generation |
| Export | canvas.captureStream + Web Audio gain-controlled mix → MediaRecorder → mp4 (or webm fallback) |
none, color-grade, halo, vhs, glow, bw, bg-blur. All branch from one fragment shader on a uPreset uniform; intensity is a slider per preset.
app/
layout.tsx // root html
page.tsx // landing
editor/page.tsx // editor route
api/deepgram-token/ // server route minting short-lived Deepgram tokens
oauth/callback/ // Tonos OAuth landing
components/
editor-shell.tsx // composition root for the editor
drop-zone.tsx // file intake
recorder-panel.tsx // in-browser re-record
compose-canvas.tsx // top canvas: reframe + shader + captions + overlays
shader-preview.tsx // WebGL2 canvas over the <video>
shader-toggle.tsx // preset chips + intensity slider
track-timeline.tsx // multi-track canvas timeline
add-track-modal.tsx // add demo / b-roll track
transcript-editor.tsx // click-to-delete word grid
transcript-toolbar.tsx // batch tools, filler-word selection
captions-overlay.tsx // canvas-rendered captions
captions-toggle.tsx
overlays-layer.tsx // text + emoji overlay rendering
overlays-toolbar.tsx
reframe-toggle.tsx // face-tracked vertical reframe
face-tracker-engine.tsx // MediaPipe landmarker host
segmenter-engine.tsx // MediaPipe selfie segmenter host
face-debug-overlay.tsx
playback-bar.tsx
export-button.tsx
tonos-connect.tsx // OAuth handshake button
tonos-draft-panel.tsx // draft a script from a prompt
tonos-rewrite-panel.tsx // rewrite selected transcript text
tonos-teleprompter.tsx // fixed-row teleprompter with Deepgram alignment
lib/
audio.ts // video → 16kHz mono Float32Array
peaks.ts // waveform peaks for timeline rendering
gaps.ts // RMS-based detection of "uh" + silent gaps
fillers.ts // filler-word classifier
suggestions.ts // batch-delete suggestions
transcript.tsx // worker orchestration + keptRanges memo + clip ops
project.tsx // multi-track project state (clips, sources)
player.tsx // PlayerProvider wiring
use-player.ts // skip-playback hook (rVFC + gap clock)
use-recorder.ts // in-browser re-record
use-export.ts // canvas + audio → mp4 via MediaRecorder
use-video-sync.ts // keep <video> aligned with project time
video-clip.ts // VideoClip type + removeSourceRange / insertSourceRange / splitClipAt / coalesce
shaders.ts // GLSL sources + compile/link helpers
shader-settings.tsx // preset + intensity context
captions.ts + captions-settings.tsx
overlays.ts + overlays-state.tsx
reframe-settings.tsx
face-tracker.tsx
segmenter.tsx
teleprompter/ // Deepgram streaming client
tonos.tsx // Tonos OAuth + draft/rewrite client
types.ts // WordToken, Range, ShaderPreset
workers/
transcribe-worker.ts // Whisper in a worker
public/sample.mp4 ships as a placeholder talking-head clip for instant demoing. Replace it with anything you record yourself; nothing in the app is hardcoded to it.
Chrome / Edge / Arc. Requires WebGL2 + requestVideoFrameCallback + MediaRecorder mp4 (or webm fallback) + AudioContext.decodeAudioData. Safari support is best-effort — the rVFC path falls back to timeupdate but isn't stress-tested.
- Everything except the Deepgram token mint + Tonos generation runs client-side. No backend storage, no upload.
- ML assets (Whisper, MediaPipe, ONNX runtime) load from CDN or local
public/models/+public/ort/(gitignored — fetched at first run). - Cuts are metadata-only; the source
.mp4is never re-encoded until export.