[Feature Request]: Latent diffusion upscaler for the Stable Diffusion autoencoder

[specblades]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Can we implement it?

Q from Twitter @RiversHaveWings:
I've trained a latent diffusion upscaler for the Stable Diffusion autoencoder (and anything you feel like feeding into it if you can tolerate a little artifacts) in collaboration with
@stabilityai
. Try the Colab written by
@nshepperd1
https://colab.research.google.com/drive/1o1qYJcFeywzCIdkfKJy7cTpgZTCM2EI4

Proposed workflow

See in colab

Additional information

No response

[Ehplodor]

Is it similar to the "scale latents" option for A1111's highres fix ? (see #2613 for example for some info)

[specblades]

Is it similar to the "scale latents" option for A1111's highres fix ? (see #2613 for example for some info)

Nope

[pbaylies]

I tested this out, wasn't hard to convert it to a stand-alone script. Note that there is a stray config file used from here - https://huggingface.co/spaces/multimodalart/latentdiffusion/blob/main/latent-diffusion/models/first_stage_models/kl-f8/config.yaml (not the CompVis repo), but just putting that file there did work.

Also added .half() in a bunch of places to use less VRAM, etc.
sdu.txt

[specblades]

I tested this out, wasn't hard to convert it to a stand-alone script. Note that there is a stray config file used from here - https://huggingface.co/spaces/multimodalart/latentdiffusion/blob/main/latent-diffusion/models/first_stage_models/kl-f8/config.yaml (not the CompVis repo), but just putting that file there did work.

Also added .half() in a bunch of places to use less VRAM, etc. sdu.txt

Can u explane for newbie how to use it?
I cant get through

[pbaylies]

@specblades probably better to start with the notebook, then, it will be friendlier to use.

[specblades]

@specblades probably better to start with the notebook, then, it will be friendlier to use.

Goal is to use it in webui, i mean
nvm, mb somebody will add it

[specblades]

Little comparison x2 upscale: native/ldsr (50 steps)/autoencoder upscale (48 steps)

The last is much sharper, but with artefacts. I like it more.
And its faster by ~10-20x
We rly need it in a1111!

[specblades]

Emad retweet it.
Hero, pls, make it work in a1111

[nagolinc]

Okay, I was able to convert @pbaylies sdu.txt into a script for automatic1111

https://gist.github.com/nagolinc/3993e7329cafab5d5bd4698977ebebcc

Before you can run it, you will need to download the two files:
https://models.rivershavewings.workers.dev/config_laion_text_cond_latent_upscaler_2.json
https://models.rivershavewings.workers.dev/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth

into your {automatic}/models/LDSR/ folder

[specblades]

Okay, I was able to convert @pbaylies sdu.txt into a script for automatic1111

https://gist.github.com/nagolinc/3993e7329cafab5d5bd4698977ebebcc

Before you can run it, you will need to download the two files: https://models.rivershavewings.workers.dev/config_laion_text_cond_latent_upscaler_2.json https://models.rivershavewings.workers.dev/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth

into your {automatic}/models/LDSR/ folder

Could you provide some kind of tutorial on how to use it?
There are no ui for change upscale ammount or steps

[specblades]

Okay, I was able to convert @pbaylies sdu.txt into a script for automatic1111

https://gist.github.com/nagolinc/3993e7329cafab5d5bd4698977ebebcc

Before you can run it, you will need to download the two files: https://models.rivershavewings.workers.dev/config_laion_text_cond_latent_upscaler_2.json https://models.rivershavewings.workers.dev/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth

into your {automatic}/models/LDSR/ folder

Many OOMs, "RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same" and other errors. Does not save the result automatically.

[arpitest]

Exception: bad file inside ./models/LDSR/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth: laion_text_cond_latent_upscaler_2_1_00470000_slim/data.pkl

The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.

[arpitest]

it start with --disable-safe-unpickle, then works only once, then restart required to work again once :( but it's good for testing, i like the result very much!
(the second run gives the "Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same" error)

could it be implemented as an upscaler, to be usable in Extras tab too, instead of being a script?
if it's a latent diffusion upscaler (as the twitter says), maybe it's enough to add an LDSR model selector to settings, to use this version of model?

[arpitest]

hmm this is also interetsing: "[Stability AI]Nov 10
That means that if you want to use it with Stable Diffusion, you take the generated "latent" and pass into the upscaler before decoding with your standard VAE."