ControlNet Channels - Restore missing RGB channel

[geroldmeisinger]

I created a ControlNet - Channels RGB (214244a32 drop=0.5 mp=fp16 lr=1e-5 epochs=2) which was trained on RGB images where each of them have one channel missing in order to restore a missing channel from an image (a special case of the Recolor ControlNet). You can either use the inference as a whole (but it changes the semantics a bit), or only use the missing channel from the inference (which doesn't look to good unfortunately). I also created an accompanying space which lets you replace the channel from an inference image. A similar evaluation using classic algorithms can be found on stackexchange. Of course it should be possible to extend this concept to color spaces like HSV or LAB and infer things like saturation or luminance etc.

• sudden change at: 227200a32(!) .. that's is pretty late
• sufficiently phenomenal convergence at: 320000a32

TODO: I think I need to try img2img

Evaluation

first is ground truth, second is inference only, third is recomposing the missing channel from inference, the image on the right is ground truth with a missing channel

Lenna (missing red)

inference only

recomposed

House2 (missing green)

inference only

recomposed

Dog2 (missing blue)

inference only

recomposed

Forest (missing red)

inference only

recomposed

Glacier (missing green)

inference only

recomposed

Fox (missing blue)

inference only

recomposed

[nevenpavlovic]

Is it possible for you to host this space again?

[geroldmeisinger]

the huggingface space? i get some runtime error and have to look into it in the next few days. keep in mind that the space only replace the channel, it's not the full stable diffusion + controlnet stack. this you have to do locally.

[nevenpavlovic]

I have an image with two channels present. So either whole reconstruction or just getting a missing channel would work for me as I can always add it as a layer to complete the image.

[geroldmeisinger]

okay, but the space only merges the layers, it does not reconstruct it in the first place. for this you need to setup stable diffusion and download the controlnet locally anyway. I'm not able to run a whole SD installation online with a free HF account.