An open agentic prompt-expansion harness for image and video generation.

Turn everyday intent into validated, model-ready prompts through a bounded AI-agent workflow.

_{Harness (loop) → Writer (LLM) → PE profile (dialect) → Adapter (optional generate)}

---

中文 · Documentation · Getting Started · Architecture · Adapters · Roadmap · Contributing

News

• 2026-08-14: omni-rewriter reconstruct observes a local clip into H3 PE. Recommended Writer is Qwen3.6-35B-A3B. This is Omni-Rewriter Replay (HF gallery).
• 2026-08 — Validate-only install: pip install omni-rewriter then omni-rewriter validate — no GPU, no Writer, no generate. Reuse the same checker in CI via WayneJin0918/Omni-Rewriter@v0.1.0.
• 2026-08 — Discussions + PE validate Action: GitHub Discussions are open, and a composite Action (action.yml) can lint PE JSON envelopes in any repo’s CI (validate only; expand ≠ generate).
• 2026-08 — H3 workflow refresh: updated the H3 video PE workflow with validated timeline, camera, dialogue, and bounded-repair rules, informed by the public MiniMax-H3 project.

About

Open Agent Harness for image/video prompt expansion (PE): schemas, orchestration, validation, and dialect render — not a bundled PE checkpoint, not a media generator.

_{Expand ≠ generate. VLM pairwise scoring is post-hoc only. SFT/RL writers are roadmap.}

How it works

This is the method. The Agent Harness owns the loop; the Writer LM is called only for Draft / Repair. Validation and render stay deterministic. Output is PE text/JSON — never media.

flowchart LR
  req[RewriteRequest] --> analyze[Analyze]
  analyze --> draft[Draft]
  draft --> validate{Validate}
  validate -->|repairable| repair[Repair]
  repair --> validate
  validate -->|ok| render[Render]
  render --> pe[PE text / JSON]
  writer[(Writer LM)] -.->|structured JSON| draft
  writer -.->|repair JSON| repair

Loading

Step	Owner	What happens
Analyze	Harness + Writer	Route video/image, read constraints/media, choose PE profile
Draft	Writer LM	Fill the profile schema as structured JSON
Validate	Harness	Deterministic checks (timeline, quotes, required fields, dialect rules)
Repair	Writer LM	Bounded retries on repairable failures; otherwise hard-fail
Render	Harness	Emit PE text for terminal / execute models (T2I, T2V, …)

Same path for CLI (omni-rewriter expand) and HTTP (POST /v1/expand). Details: architecture.

Writer LM agents

The harness talks to Writers over one contract: OpenAI-compatible chat + structured JSON. It does not ship a fine-tuned PE checkpoint. How you host the Writer is separate from which model family you pick.

How the two Writer classes connect

• Core agents (closed frontier) — GPT-5.6, Claude Opus 5, and similar. Connect via a vendor API or an OpenAI-compatible gateway (OMNI_WRITER_BACKEND_BASE_URL + model name).
• Terminal / open weights — QwenLM today; MiMo, Kimi, DeepSeek wanted. Serve locally (or on your cluster) with vLLM or SGLang, then point the same env vars at that OpenAI-compatible endpoint.

Three access modes (same protocol)

1. API — hosted frontier agents; no local GPU required for the Writer.
2. vLLM — common path for open-weight Writers (/v1/chat/completions, structured output).
3. SGLang — alternate OpenAI-compatible serve path for open Writers (also used by optional image/video generation adapters outside expand).

_{Generation adapters stay outside expand — see the compatibility matrix.}

Model ecosystem

Community board — left color = category (Video / Image / Unified); right = evidence depth (PE / adapter / unverified / wanted). Prefer small PRs with a title prefix — see CONTRIBUTING.md.

 

   

_{Use the model contribution skill. Evidence-scoped details: compatibility matrix · full backlog.}

Video RAW vs PE

Concert crash-zoom · RAW	Concert crash-zoom · PE
Kitchen whip-pan · RAW	Kitchen whip-pan · PE
Rooftop arc proposal · RAW	Rooftop arc proposal · PE

_{Current video profile: MiniMax-H3. Homepage picks: s15 concert crash-zoom, s14 kitchen whip-pan, s13 rooftop arc (from the published 15-pair set).}
H3 PE site → · Local H3 PE entry (demo → home) · Full 15-pair showcase · Compact gallery · SOURCE vs REPLAY reconstruct

Video SOURCE vs REPLAY

Observe a local clip (omni-rewriter reconstruct) into validated H3 t2va PE, then optionally replay on MiniMax-H3. Each clip is labeled Source (left) vs Omni-Rewriter (right). Compare uses the first 10s of each side. Expand ≠ generate.

10s official T2VA · left Source · right Omni-Rewriter

H3 cinematic · first 10s · left Source · right Omni-Rewriter

Seedance promo · first 10s · left Source · right Omni-Rewriter

H3 montage · first 10s · left Source · right Omni-Rewriter

SOURCE vs REPLAY gallery →

Try it (no GPU)

Lint a PE envelope. This path does not call a Writer and does not generate media.

pip install omni-rewriter
# or: uvx --from omni-rewriter omni-rewriter validate kite.json

curl -fsSL -o kite.json \
  https://raw.githubusercontent.com/WayneJin0918/Omni-Rewriter/v0.1.0/tests/fixtures/t2va_kite.json
omni-rewriter validate kite.json

In another repo’s GitHub Actions:

- uses: actions/checkout@v4
- uses: WayneJin0918/Omni-Rewriter@v0.1.0
  with:
    files: prompts/**/*.json

Expand (below) still needs an OpenAI-compatible Writer. Expand ≠ generate.

Reconstruct a clip

Turn a local short mp4 into validated H3 t2va PE. The source stays on disk.

omni-rewriter reconstruct clip.mp4 --pack-only --pack-dir /tmp/pe-pack
omni-rewriter reconstruct --from-observation docs/design/examples/observation_kite.json
omni-rewriter reconstruct clip.mp4

--pack-only needs ffmpeg, not a Writer. --from-observation needs a text Writer. A full mp4 read needs a vision Writer. Generate remains a separate adapter step.

Quick start (expand)

Gallery demos need no GPU. Preferred local path: SGLang Qwen3.6-35B-A3B (language + vision Writer) + SGLang MiniMax-H3 (~30B FL2VA). Hosted API Writers are a fallback. Expand ≠ generate — H3 is only for optional media after PE.

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[cli,server]"
cp .env.example .env

1) Local SGLang (recommended)

Terminal A — Qwen3.6-35B-A3B Writer (OpenAI-compatible chat on :8000):

export OMNI_WRITER_MODEL=Qwen/Qwen3.6-35B-A3B          # HF id or local snapshot
export OMNI_WRITER_SERVED_MODEL_NAME=Qwen/Qwen3.6-35B-A3B
# optional: OMNI_WRITER_TENSOR_PARALLEL_SIZE=4
bash scripts/serve/serve_sglang_qwen_writer.sh

Terminal B — MiniMax-H3 FL2VA via SGLang diffusion (videos API on :30010):

export OMNI_WRITER_H3_MODEL=/path/to/MiniMax-H3/FL2VA  # local H3 checkpoint
export OMNI_WRITER_H3_NUM_GPUS=8                       # match your node
bash scripts/serve/serve_sglang_h3.sh

Point Omni-Rewriter at both, then expand (and optionally generate):

export OMNI_WRITER_BACKEND_BASE_URL=http://127.0.0.1:8000/v1
export OMNI_WRITER_BACKEND_MODEL=Qwen/Qwen3.6-35B-A3B
export OMNI_WRITER_H3_BASE_URL=http://127.0.0.1:30010

omni-rewriter expand examples/requests/t2va_kite.json
omni-rewriter expand examples/requests/t2va_kite.json --output h3
omni-rewriter validate output.json
# optional generate (needs H3 up): scripts/promo/submit_h3_chunk.py … — see H3 adapters

Lint PE JSON in CI (GitHub Action)

Same checker as omni-rewriter validate (no Writer, no media):

- uses: actions/checkout@v4
- uses: WayneJin0918/Omni-Rewriter@v0.1.0
  with:
    files: prompts/**/*.json

Local multi-file helper: python scripts/validate_pe_files.py tests/fixtures/**/*.json. See Discussions #4. Marketplace listing copy: docs/pe-validate-action.md.

2) Hosted API Writer (fallback)

No local Writer GPU — use any OpenAI-compatible chat API (H3 generate can still be local):

export OMNI_WRITER_BACKEND_BASE_URL=https://api.openai.com/v1   # or any gateway
export OMNI_WRITER_BACKEND_MODEL=gpt-5.6
export OMNI_WRITER_BACKEND_API_KEY=sk-...

omni-rewriter expand examples/requests/t2va_kite.json --output h3

Checked-in requests: examples/requests/. Scripts: scripts/serve/. Details: Getting Started, H3 adapters.

Project layers

RewriteRequest
  └─ Agent Harness       analyze · draft · validate · repair · render  (= PE flow)
      └─ PE profile      H3 / Seedance / LTX-2.5 / Seedream / Qwen-Image dialect
          └─ adapter     optional vLLM / SGLang / vendor client  (generate, not expand)
              └─ eval    structural checks · RAW/PE demos under docs/

• Harness: contracts + agent loop (this release).
• Profiles: public prompt dialects for video/image generators.
• Adapters: opt-in generation clients; never called by service.expand.
• Evaluation: structure-first checks; VLM pairwise is post-hoc only.

Documentation

Guide	English	中文
Documentation index	Open	打开
Getting started	Open	打开
Architecture	Open	打开
Video prompt expansion	Open	打开
Image prompt expansion	Open	打开
Generation adapters	Open	打开
Evaluation	Open	打开

Development

python -m pip install -e ".[dev]"
ruff check .
mypy src
pytest
python -m build

Contributions are welcome across core schemas, dialects, adapters, evaluation, documentation, and future SFT/RL work. Start with CONTRIBUTING.md and ROADMAP.md.

Scope and license

Omni-Rewriter does not attempt to reproduce undisclosed closed-source behavior. It uses public contracts and reproducible examples to help the community close the gap between polished demos, public APIs, and deployable workflows. Untested runtime compatibility is labeled unverified.

Source code is licensed under Apache License 2.0. Third-party models, services, documentation, and names remain subject to their own terms. Security guidance is in SECURITY.md.