OminiControl

sudo apt-get update && sudo apt-get install git-lfs cbm ffmpeg
git clone https://huggingface.co/spaces/svjack/OminiControl && cd OminiControl && pip install -r requirements.txt
python app.py

Genshin Impact Products Examples

Image Files

References

• IMG_bat_ref.jpeg
  
• IMG_hs_card_ref.jpeg
  
• IMG_kv_card_ref.jpeg
  
• IMG_hs_ref.jpeg
  
• IMG_kv_ref.jpeg
  

Generated Images

Cards

• IMG_bat_gen.webp
  

  Prompt: A film style shot. On the moon, this item on the moon surface. The background is that Earth looms large in the foreground.

• image_hs_card_gen.webp
  

  Prompt: In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

• image_kv_card_gen.webp
  

  Prompt: In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

Dolls

• image_hs_doll_gen.webp
  

  Prompt: This item, a charming doll, nestles in a plush bed surrounded by colorful pillows in a pastel-colored bedroom. Its velvety fabric feels luxuriously soft, while its downy-stuffed limbs offer a huggable plumpness. A gentle breeze from the open window rustles sheer curtains, casting delicate light patterns on the floor. A small wooden nightstand holds a tiny lamp, casting a warm glow. The room's faint lavender scent and the distant clock's soft hum create a tranquil ambiance.

• image_kv_doll_gen.webp
  

  Prompt: This item, a charming doll, nestles in a plush bed surrounded by colorful pillows in a pastel-colored bedroom. Its velvety fabric feels luxuriously soft, while its downy-stuffed limbs offer a huggable plumpness. A gentle breeze from the open window rustles sheer curtains, casting delicate light patterns on the floor. A small wooden nightstand holds a tiny lamp, casting a warm glow. The room's faint lavender scent and the distant clock's soft hum create a tranquil ambiance.

Use LTX Video I2V to make Genshin Impact product video (https://github.com/svjack/LTX-Video)

Bat_card_LTX_Video.mp4

Bat_card_LTX_Video.mp4

Prompt: A film style shot. On the moon, this item on the moon surface. The background is that Earth looms large in the foreground.

Hs_card_LTX_Video.mp4

Hs_card_LTX_Video.mp4

Prompt: In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

Hs_doll_LTX_Video.mp4

Hs_doll_LTX_Video.mp4

Prompt: This item, a charming doll, nestles in a plush bed surrounded by colorful pillows in a pastel-colored bedroom. Its velvety fabric feels luxuriously soft, while its downy-stuffed limbs offer a huggable plumpness.

OminiControl: Minimal and Universal Control for Diffuison Transformer
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang
Learning and Vision Lab, National University of Singapore

Features

OmniControl is a minimal yet powerful universal control framework for Diffusion Transformer models like FLUX.

• Universal Control 🌐: A unified control framework that supports both subject-driven control and spatial control (such as edge-guided and in-painting generation).

• Minimal Design 🚀: Injects control signals while preserving original model structure. Only introduces 0.1% additional parameters to the base model.

Quick Start

Setup (Optional)

1. Environment setup

conda create -n omini python=3.10
conda activate omini

2. Requirements installation

pip install -r requirements.txt

Usage example

1. Subject-driven generation: examples/subject.ipynb
2. In-painting: examples/inpainting.ipynb
3. Canny edge to image, depth to image, colorization, deblurring: examples/spatial.ipynb

Generated samples

Subject-driven generation

Demos (Left: condition image; Right: generated image)

Text Prompts

• Prompt1: A close up view of this item. It is placed on a wooden table. The background is a dark room, the TV is on, and the screen is showing a cooking show. With text on the screen that reads 'Omini Control!.'
• Prompt2: A film style shot. On the moon, this item drives across the moon surface. A flag on it reads 'Omini'. The background is that Earth looms large in the foreground.
• Prompt3: In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.
• Prompt4: In a Bauhaus style room, this item is placed on a shiny glass table, with a vase of flowers next to it. In the afternoon sun, the shadows of the blinds are cast on the wall.

More results

• Try on:
• Scene variations:
• Dreambooth dataset:

Spaitally aligned control

1. Image Inpainting (Left: original image; Center: masked image; Right: filled image)

• Prompt: The Mona Lisa is wearing a white VR headset with 'Omini' written on it.
  
• Prompt: A yellow book with the word 'OMINI' in large font on the cover. The text 'for FLUX' appears at the bottom.
  

2. Other spatially aligned tasks (Canny edge to image, depth to image, colorization, deblurring)
   

   Click to show

   

   Prompt: A light gray sofa stands against a white wall, featuring a black and white geometric patterned pillow. A white side table sits next to the sofa, topped with a white adjustable desk lamp and some books. Dark hardwood flooring contrasts with the pale walls and furniture.

Models

Subject-driven control:

Model	Base model	Description	Resolution
experimental / subject	FLUX.1-schnell	The model used in the paper.	(512, 512)
omini / subject_512	FLUX.1-schnell	The model has been fine-tuned on a larger dataset.	(512, 512)
omini / subject_1024	FLUX.1-schnell	The model has been fine-tuned on a larger dataset and accommodates higher resolution. (To be released)	(1024, 1024)

Spatial aligned control:

Model	Base model	Description	Resolution
experimental / <task_name>	FLUX.1	Canny edge to image, depth to image, colorization, deblurring, in-painting	(512, 512)
experimental / <task_name>_1024	FLUX.1	Supports higher resolution.(To be released)	(1024, 1024)

Citation

@article{
  tan2024omini,
  title={OminiControl: Minimal and Universal Control for Diffusion Transformer},
  author={Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang},
  journal={arXiv preprint arXiv:2411.15098},
  year={2024}
}