Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Atomic capabilities the creation workflows compose against — pull one when you
- `/hyperframes-animation` — all animation knowledge: atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP default, plus Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU).
- `/hyperframes-keyframes` — seek-safe keyframe authoring across runtimes: GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, text trails, 3D depth; plus `hyperframes keyframes` diagnostics for surfacing and verifying rendered motion.
- `/hyperframes-creative` — non-animation creative direction: `frame.md` / `design.md` handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns.
- `/media-use` — the media OS: resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record; generate via TTS / music / image models when the catalog misses; transcribe, caption, remove backgrounds, and reuse assets across projects. One shared `scripts/audio.mjs` engine + manifest tracking; keeps search noise on disk.
- `/media-use` — the media OS (a host app's own music and sound-effect tools come first for those): resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record; generate via TTS / music / image models when the catalog misses; transcribe, caption, remove backgrounds, and reuse assets across projects. One shared `scripts/audio.mjs` engine + manifest tracking; keeps search noise on disk.
- `/hyperframes-audio` — mix the audio already placed in a composition: voiceover carve (dip a music bed only in the bands the voice occupies, static or dynamic, level match included), the effect chain (EQ, compressor, limiter, gate, saturation, delay, reverb, chorus, phaser, bitcrush), automation envelopes on volume or any effect parameter, and submix buses (`<hf-audio-group>`) that carry one chain, fader and automation clock for several tracks at once. Sourcing the audio is `/media-use`; this is what happens to it afterwards.
- `/hyperframes-cli` — CLI dev loop: `init`, `add`, `lint`, `check`, `snapshot`, `preview`, `render`, `publish`, `doctor`, `lambda` (AWS Lambda cloud rendering).
- `/hyperframes-registry` — search, install and wire registry blocks and components into compositions via `hyperframes catalog` / `hyperframes add`. Load it before hand-building any named look, effect, treatment or transition: the search ranks the whole hosted registry with nothing installed. Covers authoring a new block or component to contribute upstream.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Atomic capabilities the creation workflows compose against — pull one when you
| `/hyperframes-animation` | All animation knowledge — atomic motion rules, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU). |
| `/hyperframes-keyframes` | Seek-safe keyframe authoring across runtimes — GSAP timelines, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, 3D depth — plus `hyperframes keyframes` diagnostics for rendered motion. |
| `/hyperframes-creative` | Non-animation creative direction — `frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns. |
| `/media-use` | The media OS — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. |
| `/media-use` | The media OS (a host app's own music and sound-effect tools come first for those) — resolve any media need (BGM, SFX, image, icon, logo, voice, color grade, LUT) into a frozen local file or paste-ready block + ledger record, generate via TTS/music/image models when the catalog misses, transcribe, caption, remove backgrounds, and reuse assets across projects. One shared audio engine + manifest tracking. |
| `/hyperframes-cli` | CLI dev loop — `init`, `lint`, `check`, `snapshot`, `preview`, `render`, `publish`, `doctor`, plus HeyGen-hosted cloud rendering (`cloud render`) and AWS Lambda rendering (`lambda deploy / render / progress`). |
| `/hyperframes-audio` | Mix the audio already placed in a composition — voiceover carve (dip a music bed only in the bands the voice occupies, static or dynamic, level match included), the effect chain (EQ, compressor, limiter, gate, saturation, delay, reverb, chorus, phaser, bitcrush), automation envelopes on volume or any effect parameter, and submix buses (`<hf-audio-group>`) carrying one chain, fader and automation clock for several tracks at once. Sourcing the audio is `/media-use`. |
| `/hyperframes-registry` | Search, install and wire registry blocks and components into compositions via `hyperframes catalog` / `hyperframes add`. Load before hand-building any named look, effect, treatment or transition. Authoring a new block or component to contribute upstream. |
Expand Down
2 changes: 1 addition & 1 deletion docs/prompting/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ The installer shows a picker. Select the **core skills** below — every project
| `/hyperframes-animation`| All animation — motion rules, scene blueprints, transitions, and the runtime adapters (GSAP, Lottie, Three.js, Anime.js, CSS, WAAPI, TypeGPU) |
| `/hyperframes-creative` | Creative direction — design spec, palettes, typography, narration, beats |
| `/hyperframes-cli` | Dev-loop CLI — `init`, `lint`, `check`, `preview`, `render`, `doctor` |
| `/media-use` | Media OS — TTS voiceover (`tts`), `transcribe`, `remove-background`, plus BGM / SFX / image resolution |
| `/media-use` | Media OS — TTS voiceover (`tts`), `transcribe`, `remove-background`, plus BGM / SFX / image resolution (a host app's own music and sound tools come first) |
| `/hyperframes-registry` | Search the catalog before hand-building a look; install via `hyperframes add` |
| `/hyperframes-keyframes`| Seek-safe keyframe authoring across runtimes, plus `hyperframes keyframes` diagnostics |
| `/general-video` | The general authoring workflow — multi-scene pieces, reels, montages, remixes, and the home of **companion mode**; the fallback when no workflow below fits |
Expand Down
22 changes: 21 additions & 1 deletion packages/cli/src/media-use/lib/heygen-cli.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,11 @@ function classifyHeygenErrorResult(err) {
}

const version = firstSemver(text);
if (version && versionLessThan(version, HEYGEN_MIN_VERSION)) {
// A CLI older than the --headers flag rejects it before printing any version.
if (
(version && versionLessThan(version, HEYGEN_MIN_VERSION)) ||
lower.includes("unknown flag: --headers")
) {
return { code: "outdated", message: HEYGEN_OUTDATED_MESSAGE };
}

Expand Down Expand Up @@ -98,6 +102,22 @@ const pendingFailureTracking = new Set();
// API, move this state into a per-resolve context before reusing that path.
let pendingRemediation = null;

const TOOL_WORDS = { bgm: "music", sfx: "sound-effect" };

/** A music or sound-effect resolve miss after the heygen CLI was missing or too old: what is missing, the host
* app's own tool, the fix. Other types keep the generic miss. */
export function heygenMiss(type, { code }) {
const tool = TOOL_WORDS[type];
if (!tool) return null;
const outdated = code === "outdated";
const state = outdated ? `older than v${HEYGEN_MIN_VERSION}` : "not installed";
return {
code: outdated ? "heygen_cli_outdated" : "heygen_cli_missing",
fix: outdated ? HEYGEN_UPDATE_COMMAND : HEYGEN_INSTALL_COMMAND,
error: `${type} needs the heygen CLI, which is ${state}: use your host app's own ${tool} tool if it has one, or ${outdated ? "update" : "install"} the CLI.`,
};
}

export function consumeHeygenRemediation() {
const remediation = pendingRemediation;
pendingRemediation = null;
Expand Down
14 changes: 14 additions & 0 deletions packages/cli/src/media-use/lib/heygen-cli.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
HEYGEN_NOT_FOUND_MESSAGE,
HEYGEN_OUTDATED_MESSAGE,
heygenMiss,
reportHeygenFailure,
} from "./heygen-cli.mjs";

Expand Down Expand Up @@ -299,3 +300,16 @@ test("flushHeygenFailureTracking waits for a pending report before resolving", a
test("flushHeygenFailureTracking resolves immediately when nothing is pending", async () => {
await flushHeygenFailureTracking();
});

test("only a music or sound-effect miss names the heygen CLI and the host app's own tool", () => {
for (const type of ["image", "icon", "voice", "video"])
assert.equal(heygenMiss(type, { code: "not_found" }), null, type);
assert.equal(heygenMiss("bgm", { code: "not_found" }).code, "heygen_cli_missing");
const sfx = heygenMiss("sfx", { code: "outdated" });
assert.equal(sfx.code, "heygen_cli_outdated");
assert.match(sfx.error, /host app's own sound-effect tool/);
});

test("a CLI that rejects --headers without printing a version is outdated", () => {
assert.equal(classifyHeygenErrorCode({ stderr: "Error: unknown flag: --headers" }), "outdated");
});
17 changes: 11 additions & 6 deletions packages/cli/src/media-use/resolve.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ import {
HEYGEN_MIN_VERSION,
HEYGEN_UPDATE_COMMAND,
consumeHeygenRemediation,
heygenMiss,
firstSemver,
flushHeygenFailureTracking,
versionLessThan,
Expand Down Expand Up @@ -530,10 +531,16 @@ async function run() {
});
// brand stays local: no frame.md/design.md -> upsell the HyperFrames design
// flow rather than reporting a generic miss (B5).
const msg =
const ownFailure =
providerFailure instanceof BundledSfxAssetsError ||
providerFailure instanceof FfBinarySettingError
? providerFailure.message
providerFailure instanceof FfBinarySettingError;
const heygen = consumeHeygenRemediation();
const miss = heygen && !ownFailure ? heygenMiss(type, heygen) : null;
const typed = providerFailure instanceof BundledSfxAssetsError ? providerFailure : miss;
const msg = ownFailure
? providerFailure.message
: miss
? miss.error
: type === "brand"
? "no brand spec found — add a frame.md or design.md (colors/font/logo) to this project. Run the HyperFrames design flow to create one; brand tokens are read locally for deterministic rendering."
: args.provider
Expand All @@ -543,9 +550,7 @@ async function run() {
console.log(
JSON.stringify({
ok: false,
...(providerFailure instanceof BundledSfxAssetsError
? { code: providerFailure.code, fix: providerFailure.fix }
: {}),
...(typed ? { code: typed.code, fix: typed.fix } : {}),
error: msg,
}),
);
Expand Down
107 changes: 87 additions & 20 deletions packages/cli/src/media-use/resolve.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@ import { execFileSync, spawn, spawnSync } from "node:child_process";
import { appendRecord, findByPrompt, readManifest } from "./lib/manifest.mjs";
import { regenerateIndex } from "./lib/index-gen.mjs";
import { getProvider } from "./lib/providers.mjs";
import { HEYGEN_NOT_FOUND_MESSAGE } from "./lib/heygen-cli.mjs";
import {
HEYGEN_INSTALL_COMMAND,
HEYGEN_NOT_FOUND_MESSAGE,
HEYGEN_UPDATE_COMMAND,
} from "./lib/heygen-cli.mjs";
import { freezeLocalFile } from "./lib/freeze.mjs";
import { cachePut, cacheGet, importFromCache } from "./lib/cache.mjs";
import { validateCubeFile } from "./lib/cube-validate.mjs";
Expand Down Expand Up @@ -244,6 +248,69 @@ test("explicit local bundled SFX resolution does not advise installation", () =>
}
});

test("music without the HeyGen CLI fails naming it and the host app's own tool", () => {
setup();
const result = spawnResolve(
["--type", "bgm", "--intent", "calm piano", "--project", tmp, "--json"],
{
env: { HOME: tmp, PATH: tmp },
},
);
assert.equal(result.status, 1, result.stderr);
const parsed = JSON.parse(result.stdout);
assert.equal(parsed.ok, false);
assert.equal(parsed.code, "heygen_cli_missing");
assert.equal(parsed.fix, HEYGEN_INSTALL_COMMAND);
assert.match(parsed.error, /needs the heygen CLI, which is not installed/);
assert.match(parsed.error, /host app's own music tool/);
cleanup();
});

test("a sound effect nothing bundled matches fails the same way without the HeyGen CLI", async () => {
setup();
// The local media index answers 404, so the miss never depends on the network.
const server = createServer((_req, res) => {
res.writeHead(404);
res.end();
});
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
try {
const result = await spawnResolveAsync(
["--type", "sfx", "--intent", "dog barking", "--project", tmp, "--json"],
{
env: {
HOME: tmp,
PATH: tmp,
HYPERFRAMES_REGISTRY: `http://127.0.0.1:${server.address().port}`,
},
},
);
assert.equal(result.status, 1, result.stderr);
const parsed = JSON.parse(result.stdout);
assert.equal(parsed.code, "heygen_cli_missing");
assert.match(parsed.error, /host app's own sound-effect tool/);
} finally {
server.close();
cleanup();
}
});

test("music with an outdated HeyGen CLI fails naming the update", () => {
setup();
const binDir = writeFakeHeygen('echo "heygen v0.1.5 does not support --headers" >&2', 1);
const result = spawnResolve(
["--type", "bgm", "--intent", "calm piano", "--project", tmp, "--json"],
{
env: { HOME: tmp, PATH: binDir },
},
);
assert.equal(result.status, 1, result.stderr);
const parsed = JSON.parse(result.stdout);
assert.equal(parsed.code, "heygen_cli_outdated");
assert.equal(parsed.fix, HEYGEN_UPDATE_COMMAND);
cleanup();
});

test("human bundled fallback prints the install hint once", () => {
setup();
const result = spawnResolve(["--type", "sfx", "--intent", "whoosh", "--project", tmp], {
Expand Down Expand Up @@ -801,9 +868,12 @@ test("missing required args exits 2", () => {

test("--json returns error JSON on stub provider failure", () => {
setup();
// A HeyGen CLI that finds nothing: the miss is the generic one on every machine, with or without a real CLI.
const binDir = writeFakeHeygen(`echo '{"data":[]}'`);
try {
runResolve(["--type", "bgm", "--intent", "stub fail", "--project", tmp, "--json"], {
stdio: "pipe",
env: { HOME: tmp, PATH: binDir },
});
assert.fail("should have exited");
} catch (err) {
Expand Down Expand Up @@ -1280,26 +1350,23 @@ async function captureResolveEvent({ provider, type = "bgm", intent }) {
// their real email into this test's local-server payload despite HOME
// being sandboxed (HEYGEN_CONFIG_DIR, not HOME, resolves the credentials
// path). Every other test in this file keeps its untouched default env.
runResolve(["--type", type, "--intent", intent, "--project", tmp, "--json"], {
env: {
DO_NOT_TRACK: "0",
HYPERFRAMES_NO_TELEMETRY: "0",
CI: "",
NODE_ENV: "test",
HOME: sandboxHome,
HEYGEN_CONFIG_DIR: join(sandboxHome, ".heygen"),
MEDIA_USE_TELEMETRY_HOST: `http://127.0.0.1:${port}`,
// Async, so this process serves the POST while the child waits on it: a blocking spawn froze the server
// until the child gave up on its telemetry timeout, and the event arrived or not by luck.
const run = await spawnResolveAsync(
["--type", type, "--intent", intent, "--project", tmp, "--json"],
{
env: {
DO_NOT_TRACK: "0",
HYPERFRAMES_NO_TELEMETRY: "0",
CI: "",
NODE_ENV: "test",
HOME: sandboxHome,
HEYGEN_CONFIG_DIR: join(sandboxHome, ".heygen"),
MEDIA_USE_TELEMETRY_HOST: `http://127.0.0.1:${port}`,
},
},
});

// runResolve blocks synchronously (execFileSync) until the child exits, which
// pauses this process's own event loop for that whole span -- the child's
// request to our local server sits accepted-but-unprocessed in the kernel
// backlog until control returns here. Poll briefly to let the event loop
// drain it rather than asserting before the server has had a turn to run.
for (let i = 0; i < 100 && received.length === 0; i++) {
await new Promise((resolve) => setTimeout(resolve, 20));
}
);
assert.equal(run.status, 0, run.stderr);
} finally {
await new Promise((resolve) => server.close(resolve));
rmSync(sandboxHome, { recursive: true, force: true });
Expand Down
4 changes: 4 additions & 0 deletions scripts/generate-skill-module-copies.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ export const skillModuleCopies = [
"skills/media-use/audio/scripts/lib/bgm-volume.mjs",
workflows.map((skill) => `skills/${skill}/scripts/lib/bgm-volume.mjs`),
],
[
"skills/media-use/audio/scripts/lib/host-audio.mjs",
workflows.map((skill) => `skills/${skill}/scripts/lib/host-audio.mjs`),
],
[
"skills/hyperframes/scripts/lib/frame-packets-core.mjs",
[...workflows, "general-video"].map(
Expand Down
22 changes: 11 additions & 11 deletions skills-manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,19 @@
"files": 144
},
"faceless-explainer": {
"hash": "fd247c01901c395c",
"files": 27
"hash": "3459707d807fdde5",
"files": 28
},
"figma": {
"hash": "f8cf9f2c78ea8374",
"files": 2
},
"general-video": {
"hash": "0fdb51f01b751cdb",
"hash": "7194a47a82647a88",
"files": 5
},
"hyperframes": {
"hash": "75dfebd2be1e95bb",
"hash": "589ae40d15628917",
"files": 29
},
"hyperframes-animation": {
Expand Down Expand Up @@ -54,24 +54,24 @@
"files": 1
},
"media-use": {
"hash": "9b98931fc057f125",
"files": 107
"hash": "b07cf8ba10a8ac7b",
"files": 109
},
"motion-graphics": {
"hash": "6db77031ad839f6a",
"files": 23
},
"music-to-video": {
"hash": "5b365022211eaf0c",
"hash": "b775f27b16efb1f2",
"files": 169
},
"pr-to-video": {
"hash": "39e2b3e5613c5eec",
"files": 33
"hash": "b38d166d7328e4cb",
"files": 34
},
"product-launch-video": {
"hash": "87a8a6895cb90865",
"files": 34
"hash": "d78ebd2c45fa8bec",
"files": 35
},
"remotion-to-hyperframes": {
"hash": "d02be65b7f3f4a63",
Expand Down
Loading
Loading