feat(media-use): transcription (parakeet), transcript-cut, audio-duck

[miguel-heygen]

Editing tools that didn't exist before: transcript-driven cutting (word-timestamp edits -> frame-accurate ffmpeg cuts), declarative auto-duck (speech spans -> GSAP volume envelope), and Parakeet-first local transcription (token->word merge, whisper.cpp fallback).

Stack: media-use v2 (5/5, top).

---

📚 Stack — media-use v2 (split from #1923, merge bottom-up)

1. fix(producer): mix muted browser media as silent for preview-render parity #1969 · producer: muted preview↔render parity
2. feat(cli): parakeet ASR engine for transcribe + HYPERFRAMES_PYTHON override #1970 · cli: Parakeet transcribe engine
3. feat(studio): non-destructive crop + cross-project asset view #1971 · studio: non-destructive crop + asset view
4. feat(media-use): v2 media OS core + retire hyperframes-media #1972 · media-use core + retire hyperframes-media
5. feat(media-use): transcription (parakeet), transcript-cut, audio-duck #1973 · media-use edit tools (transcript-cut, audio-duck, Parakeet)

[miguel-heygen]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat(media-use): transcription (parakeet), transcript-cut, audio-duck #1973 👈 (View in Graphite)
• feat(media-use): v2 media OS core + retire hyperframes-media #1972
• feat(studio): non-destructive crop + cross-project asset view #1971
• feat(cli): parakeet ASR engine for transcribe + HYPERFRAMES_PYTHON override #1970
• fix(producer): mix muted browser media as silent for preview-render parity #1969
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[vanceingalls]

Verdict: comment (minor runtime-interop notes; nothing blocking)

A capable trio of scripts, and I confess the evidence rewards patient attention. The Parakeet subprocess contract is guarded, the ffmpeg pipeline is atomic to a tmpdir, and the ducker is a well-behaved code-emitter — no red flags on the interop lens. A handful of observations, in descending order of consequence.

---

1. Parakeet on PATH — install docs vs invocation reality

skills/media-use/scripts/transcribe.mjs:39-42 prints the install instruction as:
uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx

But hasBin("parakeet-mlx", ["--help"]) at line 65 and execFileSync("parakeet-mlx", ...) at line 78 spawn the binary from PATH — no VIRTUAL_ENV propagation, no ~/.venvs/parakeet/bin/parakeet-mlx fallback. A user who follows the printed instructions verbatim installs into an isolated venv, exits the shell, and parakeet-mlx is no longer on their PATH. hasBin returns false, auto falls back to whisper silently, and the user is quietly on the slower/less-accurate engine forever. With --engine parakeet they at least see the error at line 130 pointing them back at the install command — cold comfort if they already ran it.

Two low-cost remedies, either fine: (a) print activation instructions alongside install ("...then add ~/.venvs/parakeet/bin to your PATH"), or (b) probe ~/.venvs/parakeet/bin/parakeet-mlx as a fallback location before falling back to whisper. I lean (a) since (b) hard-codes the recommended path.

2. runWhisper copies whatever hyperframes transcribe produces — shape drift is invisible here

transcribe.mjs:105 does copyFileSync(produced, outPath), then a best-effort JSON parse just to count words for the report. The whisper.cpp CLI's transcript.json schema is not enforced against the Parakeet-normalized { text, words: [{text,start,end}] } shape. Happily, lib/words.mjs:normalizeWords accepts either w.text or w.word and drops anything missing finite start/end — so the mismatch degrades gracefully at the consumer. Worth stating explicitly in the file header ("output shape is contract for downstream normalizeWords") so the next author doesn't tighten mergeTokensToWords in a way that diverges.

3. Non-atomic output writes on both entry points

transcribe.mjs:88 writes outPath directly via writeFileSync; transcript-cut.mjs:139-142 concats to outPath via ffmpeg with no temp+rename dance. If the process is SIGKILL'd mid-write (Ctrl-C on a long ffmpeg encode, OOM kill, laptop lid closed), outPath is left half-written and future invocations that skip existing outputs will consume the corrupt file. The prior tmpdir usage inside transcript-cut.mjs already sets up the scaffolding — writing the ffmpeg concat to tmpDir/final.mp4 and renaming to outPath after ffmpeg exits 0 is a one-line change. Non-blocking, but the correctness delta is real if these scripts get scripted into pipelines.

4. --copy keyframe drift is a console.error warning, not a JSON error

transcript-cut.mjs:150-155 diagnoses keyframe-snapping drift after the fact and warns to stderr. In --json mode (line 165), the JSON still reports ok: true while stderr carries the warning. A pipeline consumer parsing stdout JSON has no signal that the cut is unreliable. Consider surfacing the drift in the JSON payload ({ ok: true, warning: 'keyframe_drift_seconds: ...' }) so downstream automation can gate on it.

5. audio-duck.mjs emits GSAP with silent selector + zero-base-volume failure modes

The ducker is a code emitter, not a mix-time processor — it prints tl.to("#bgm", { volume: 0.25, ... }) lines that a downstream composition consumes. Two silent failures worth acknowledging in the help text or the emitted comment:

• GSAP tl.to on a missing selector is a no-op. If --target "#bgm" typos or the element is removed later, the emitted timeline runs cleanly and produces zero ducking — no error at generate time, no error at render time.
• readBaseVolume (line 96-106) reads data-volume off the target; if the composition sets data-volume="0" (muted), baseVolume=0 and ducked = 0 * 0.25 = 0 — the whole envelope is flat-zero, but the emitted tl.to lines all look plausible. A console.error warning when baseVolume === 0 would catch this.

The "ponytail" self-note on the regex at line 100 is correct in scope; no concerns there.

6. npx hyperframes transcribe cold-cache install

transcribe.mjs:99 shells out via npx hyperframes transcribe .... If hyperframes isn't in the local npx cache and the user is offline (or on a locked-down runner), npx will attempt a network install and hang or fail. The 30-minute timeout: 1_800_000 means this could burn the whole timeout budget on a network probe rather than the transcription itself. In practice inside skills/media-use the CLI is a peer dep, but worth confirming hyperframes resolves locally before shelling to npx.

7. Media OS consumption — clean

Verified the three .mjs entry points import only from ./lib/* (the four new files + pre-existing words.mjs, telemetry.mjs). No lingering hyperframes-media imports; #1972's retirement holds. skills-manifest.json diff is the expected media-use hash bump (4e07a35f → 21951022f) + file count 87 → 97, no new skill entries and no namespace collisions.

8. Determinism — clean

Parakeet-MLX JSON output is deterministic per model+input; mergeTokensToWords, compileCutList, speechSpans, duckKeyframes are pure and go through round3 at every boundary. ffmpeg -c copy concat is byte-deterministic. Producer render cache is safe from this PR.

---

The interop surface is small, well-scoped, and behaves the way its help text says it does. The rough edges above are ergonomic and pipeline-safety concerns, not correctness ones — none warrant blocking. Approving with the notes; the atomic-write and keyframe-drift JSON improvements are the two I'd most like to see land.

Review by Via (runtime-interop lens)

[jrusso1020]

APPROVE — part of the 5-PR media-use stack (#1969–#1973), two-lens verification. Via (runtime-interop) + a test/observability pass both clear all five, no blockers.

Verified:

• Headline "addressed" items confirmed: #1972's orphan hyperframes-media strings fixed at both call sites (sfx.mjs:125, tts.mjs:184; the only surviving ref is SKILL.md's historical changelog line, intentional); #1973's atomic writes in place on both entry points (transcribe.mjs temp+renameSync, transcript-cut.mjs encode-to-.part+rename), and the parakeet PATH→venv fallback (resolveParakeet()) is resolved.
• Test coverage solid: #1969 mutedParity.test.ts (all mute combos), #1970 parakeet.test.ts (token-merge boundaries), #1971 crop/affordance/asset-strip tests, #1973 ~0.8 test:source ratio across 4 new files.

Non-blocking follow-ups (Via-accepted nits, worth tracking, NOT gating): #1970 observability (--model ignored on the parakeet path, subprocess stderr discarded on success, no first-run model-download cue, HYPERFRAMES_PARAKEET skips the --version gate); #1973 --copy keyframe-drift warning only on stderr, not in --json.

Merge via the normal Graphite path (bottom-up, no admin-merge). Layered on Via.

— Rames Jusso

[miguel-heygen]

Landed on main (media-use v2 stack, ac0153c). The Graphite merge queue had retargeted this PR's base to a synthetic graphite-base/* branch, so GitHub couldn't auto-close it; the slice was merged to main directly. Closing.