IC-Light V2 (Flux-based IC-Light models)

[lllyasviel]

Note that this post is a work in progress (wip). Maybe I will edit it a lot recently.

---

IC-Light V2 is a series of Flux-based IC-Light models with 16ch VAE and native high resolution.

We plan to have several model variations:

• foreground conditioned model, emphasizing the preservation of input image details (this model).
• foreground conditioned model, emphasizing the ability to make large modifications like processing low-light images and change hard shadows (not released yet).
• foreground and background conditioned model (not released yet).
• integration with environment HDRIs (not released yet).

News

See also alternative model for stronger illumination variations and modifications here: #109

Foreground Conditioned Model

https://huggingface.co/spaces/lllyasviel/iclight-v2

Some notes:

Thanks to 16ch VAE, native high resolution, and better training methods, the detail preservation capability is much higher than SD1.5.

For example:

If you overlay the inputs/outputs, you will see the level of detail preservation, like:

another one:

Another important difference to SD1.5 is the ability to process stylized images. The SD15 IC-Light sometimes makes too large modification to image styles, like:

As you can see, the SD15 iclight makes the input drawing a photo-like image. The effect is still interesting but may not be the desired one in some cases.

Below is another attempt to add “surrealist painting of” to prompt but still cannot get a painting:

These are from new model:

(3 results with same seed 12345)

It depends on the taste of different people.

But at least the current model is finally able to give you a painting when you do not want a photo..

Examples, all with seed 12345

More info about other variations to be continued ...

Special thanks

Special thanks to FAL for the help with computation resources on many internal experiments. (I will edit this "special thanks" section later to add more items...)

[bsdesigner]

Super! Thanks for your work!

[slmonker]

still not working well on full body photo......

[lllyasviel]

In case people ask: You can only use the hf space to test for now. More inference codes and weights may be released later.

However, unlike V1, the license will be non-commercial only.

Also this time we have several additional credits to add to thank for help with training computation power. I will edit this post again later (maybe today) for this. done.

[rongtian0927]

I used SD1.5 of iclight and Krita to create a workmap where I could draw lightfades freely, and I could control the CFG value to achieve a controlled lighting effect. I think this method of use is the real future, and it should be more suitable for professional business applications. Looking forward to flux_iclight!

Hi，how can I learn from you about the iclight + krita use to generate a photo？

[dkqjhoifh]

https://discord.com/channels/1282119084145643520/1282119084749619334/1300276485449322627
What you need is here！

[janekm]

In case people ask: You can only use the hf space to test for now. More inference codes and weights may be released later.

However, unlike V1, the license will be non-commercial only.

Also this time we have several additional credits to add to thank for help with training computation power. I will edit this post again later (maybe today) for this. done.

Regarding license... will fal have a commercial license for their API? That could be a solution for many who want to use it commercially

[CJohnDesign]

Oh so Fal is offering this and we get it from them? How would we use that in ComfyUI?

[creamlabsAI]

Cant wait to get FAL access!

[S4f3tyMarc]

Thanks you! It was only 2 days ago I was thinking I really hope we get IC-Light for Flux. You are a legend :)

[ynie]

I hope we can get sd3.5 soon

[danbochman]

Awesome work! Any tips on how to handle cases where some color preservation is needed? For example here the colors of the graphic print on the tank-top change too much:

[oxygen00004]

Very powerful!

[toyxyz]

Thanks for the awesome work!

[KirnMeng]

So awesome! It will be great when it can be applied to ComfyUI!

[bs001l]

+1

[zichanghu]

Great job! Hope sometime I can use API to direct use it.

[joel-galleri5]

is it applicable for comfyui rn?

[formfollows]

great work! on which flux model is it going to base on? schnell would be amazing in regards to its commercial license :)

[jbhwong]

Awesome!
I have a question, are these IC-light models base on ControlNet or modified flux base models whose input channels are modified to support other inputs such as foreground channels?
Looking forward to your reply! Thanks.

[SanPy]

I found that in the original picture, for example, a bottle of shampoo, the Chinese characters on the bottle become distorted

[kou35]

有文字的图片，无论是背景生成的，还是需要打光的物体，文字的处理感觉都是比较差的，不是很稳定

[SanPy]

是的，但是我发现提高原图片的分辨率可以解决这个问题

[xxxbrokenboi-lkk]

解决了这个问题吗，同志是否有图片案例可分享，我想知道你的解决了这个问题是到了什么程度

[SanPy]

试试4k以上分辨率，但是这样对硬件要求就会高了，时间也会长

[LinhaiSmallYu]

后处理调用一下restore detail试试

[txhno]

Amazing. Bit of a bummer on it being non-commercial tho, but great work nonetheless.

[FurkanGozukara]

awesome work

can you also add that light direction circle like comfyui had to the gradio demo before releasing local version of the app?

thank you so much

[fabiopolimeni]

Or just add a way to provide a custom the light map

[bragilee]

very look forward to it as well!

[mikebilly]

Any progress?, please update us with the status

[vlrevolution]

Tuning in for this to use it in comfyui :)

[studiokevinabanto]

Amazing work, waiting the comfyui workflow

[lllyasviel]

Hi iclight users,

We detected a mistake in the previous implementation causing checkbord artifacts and degraded quality. A fix is already applied to the demo. Here is the comparison:

Before:

(influenced by the mistake causing checkboard artifacts in the foreground and quality degradation in the background)

After:

(bugs fixed, the current demo)

And by the way we will have more demo variations soon

[ykj467422034]

Hello，when to release the model? Very expect!

[V1sionVerse]

Nice, the outputs from the new version look much cleaner!
Can't wait to run it locally.

[dan4ik94]

We detected a mistake in the previous implementation causing checkbord artifacts and degraded quality. A fix is already applied to the demo.

The previous version with artifacts was better in many scenarios. Would be nice to have a checkbox/select to choose if we want the fixed or "wrong" one.

[vlrevolution]

Wow the fix is amazing, I was wondering why it was smudged before :) thank you so much for the amazing work! How can we use this at scale as an API?

[fabiopolimeni]

fal.ai already has it available as an API

[LiangChengJZ]

666 nb 期待完全体

[Guiyangbiao]

V1 version will lose the colour how to change it.

[dizmaja]

Hi, is it possible to make higher quaility images, biggest I made was 1300x1300, and that is pretty low. thx

[szhang963]

Hi， this is an excellent work.
Can we use a specified image as a background to generate illumination and shadow for the inserted object and output a harmonious image?
Thanks.

[Prajwal-Mhatre]

where is the research paper? i cannot find anything on web regarding this work.

[ChieVFX]

Great work! Looks very useful from my tries.
Looking forward to using this in Comfyui!

[Latentnaut]

Wow, looking forward the release for Comfy, very promising.

[abhayrrd]

this is a good work

[bbt9g]

any news?

[vlrevolution]

Checking in as well