vibe scene build: I2V backdrop integration (motion video instead of still + Ken-Burns)

[kiyeonjeon21]

Why

ROADMAP Phase 4 lists "I2V backdrop integration" as an open item. Today every backdrop in vibe scene build is a still image animated with kenBurnsTween() — pan/zoom only. Adopting image-to-video providers (Runway, Kling, Veo, fal.ai) would unlock real motion backdrops without losing the cache or the per-beat YAML cue model.

State of code (2026-04-29)

• packages/cli/src/commands/_shared/scene-html-emit.ts:254 — kenBurnsTween() is the only motion source, applied at lines 344, 383, 435.
• packages/cli/src/commands/scene.ts:889-920 — backdrop asset generation is image-only (T2I providers).
• I2V providers exist in @vibeframe/ai-providers (used today by vibe gen vid -i), so the provider plumbing is already there — what's missing is the per-beat dispatch + HTML emit branch.

Scope (sketch — design doc probably needed)

• Per-beat YAML cue: backdrop.video: <prompt> or backdrop.kind: video opting into I2V
• Routing flag: --video-backdrop-provider <runway|kling|veo|fal> (or extend --image-provider)
• Two-step dispatch: T2I (still) → I2V (motion). Cache key must cover both stages.
• HTML emit: when backdrop is a video, emit <video> with __hf.media registration instead of <img> + kenBurnsTween()
• Length matching: video backdrop must match beat duration (loop, trim, or generate-to-length)
• Cost: I2V is much pricier than T2I — --describe envelope and budget gates need updating
• Fallback: if I2V fails, degrade to still + Ken-Burns rather than aborting the whole build

Out of scope

• T2V (text-to-video without an image) — separate concern
• Multi-provider T2I (vibe scene build: multi-provider T2I routing (Gemini + Grok) + fix help text #202) — tracked separately

Recommendation

Big enough to warrant a docs/design/ doc + plan PR series before implementation, similar to Plan G/H. Probably 4-6 PRs.

Reference

• ROADMAP.md Phase 4 "Open items in Phase 4 (v0.61+ candidates)"
• Existing I2V usage: packages/cli/src/commands/ai-video.ts

[kiyeonjeon21]

Maintainer note: part of the surrounding video-provider plumbing now exists, but this issue is not done.

Current main has per-beat video cue handling and --video-provider support for build assets. That gives us generated assets/video-<beat>.mp4 files and pending-job tracking for providers like Seedance, Grok, Kling, Runway, and Veo.

The missing part is the actual “video backdrop” integration described here:

• the generated video asset is not yet wired into the scene composition as the visual backdrop
• root sync currently handles scene HTML plus narration/music audio, not visual video backdrops
• the composer prompt / HTML contract still needs to define when to mount assets/video-<beat>.mp4 instead of assets/backdrop-<beat>.png
• fallback behavior from video to still backdrop still needs a defined policy

So this should stay open, but the scope should be read as “wire generated or referenced video assets into the visual scene/background path”, not “add video providers from scratch”.