Skip to content

v0.16
Compare
Choose a tag to compare
@Sxela Sxela released this 31 Oct 14:52
· 1 commit to v0.16-AGPL since this release
v0.16
910895e

Changelog:

  • add prompt weight parser/splitter
  • update lora parser to support multiple prompts. if duplicate loras are used in more than 1 prompt, last prompt lora weights will be used
  • unload unused loras
  • add multiple prompts support
  • add max batch size
  • add prompt weights
  • add support for different prompt number per frame
  • add prompt weight blending between frames
  • bring back protobuf install
  • add universal frame loader
  • add masked prompts support for controlnet_multi-internal mode
  • add controlnet low vram mode
  • add masked prompts for other modes
  • fix undefined mask error
  • fix consistency error between identical frames
  • add ffmpeg deflicker option to video export (dfl postifx)
  • export video with inv postfix for inverted mask video
  • add sd_batch_size, normalize_prompt_weights, mask_paths, deflicker_scale, deflicker_latent_scale to gui/saved settings
  • fix compare settings not working in new run
  • fix reference controlnet not working with multiprompt
  • disable ffmpeg deflicker for local install
  • fix torchmetrics version thx to tomatoslasher
  • fix pillow error
  • fix safetensors error

Multiple prompts

You can now use multiple prompts per frame. Just like this:

{0:['a cat','a dog']}

In this case with no weights specify it will give each prompt a weight of 1.

You can speciffy weights like this: {0:['a cat:2','a dog:0.5']}

The weights should be at the end of the prompt.

normalize_prompt_weights: enable to normalize weights to add up to 1.

For example this prompt {0:['a cat:2','a dog:0.5']} with normalize_prompt_weights on will effectively have weights {0:['a cat:0.8','a dog:0.2']}

Prompt weights can be animated, but the weight will be applied to the specific prompt number, not exact text. So {0:['prompt1:1', 'prompt2:0'], 5: ['prompt1:0', 'prompt3:1']} will blend the weights but not the prompts, so you will have prompt1 until frame5, then it will be replaced with prompt3, but the weights will be animated, so that a prompt for a frame between 0 and 5 will look like ['prompt1:0.5', 'prompt2:0.5']

You can have different number of prompts per frame, but the weights for prompts missing in a frame will be set to 0

For example, if you have:

{0:['a cat:1', 'a landscape:0'], 5: ['a cat:0', 'a landscape:1'], 10:['a cat:0', 'a landscape:1', 'a galaxy:1']}

'a galaxy' prompt will have 0 weight for aall frames where it's missing, and will have weight 1 at frame 10

Each additional prompt adds +50% to vram usage and diffusion render times.

Masked prompts

You can now use masks for your prompts. The logic is a bit complicated, but I hope you'll get the idea.

You can use masks if you have more than one prompt.

The first prompt is always the background prompt, you don't need a mask for it.
If you decide to use masks, you will need to provide them for every other prompt other than the 1st one. Each next prompt+mask will be placed on top of the previous, only white areas of the mask will be preserved. For example, if your 2nd prompt mask is completely covering the 1st prompt mask, you will not see the 1st prompt in the output as it will be covered by the 2nd prompt mask completely.

You need to specify path to your mask frames/video in the mask_paths variable. for 2 prompts you will need 1 mask, for 3 prompt - 2 masks, etc.

Leave mask_paths=[] to disable prompt masks. Enabling prompt masks will effectively disable prompt weights.

Max_batch_size

By default oyur image is diffused with a batch = 2, consisting of conditioned and unconditione images (positive and negative prompt). When we add more prompts, we to diffuse more images, one extra image per extra prompt.

Depending on your gpu vram, you can decide to increase batch size to process more than 2 prompts at a time.

You can set batch size to 1 to reduce VRAM usage even with 1 prompt.

Controlnet low vram fix

Enable to juggle controlnets from cpu to gpu each call. Is very slow, but saves a lot of vram. Right now all controlnets are offloaded and loaded to gpu ram once per frame, so that they are only kept on GPU during diffusion.

With controlnet_low_vram=True all controlnets will stay offloaded to cpu and only be loaded to gpu when being called during diffusion, then offloaded back to cpu, each diffsuion step.

Fixes

Unused loras should now be kept unloaded in a new run that doesn't use loras.

Assets 3
Loading