-
Couldn't load subscription status.
- Fork 433
vae tiling improvements: encoding support and adaptative overlap #484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
9edc59f to
af0491b
Compare
6f49af0 to
6f96903
Compare
Nice! It would also be interesting to see larger tile sizes. Depending on VAE, larger tiles would still fit into the memory budget and would speed things up. |
5495c93 to
e9a8d17
Compare
e9a8d17 to
d103632
Compare
|
I am not familiar with AI programming, and this is a machine translation, so please ignore me if I say something strange. The However, the brightness issue (issue #588) seems to remain. |
I'm sorry, I don't quite understand what you mean.
You mean this PR doesn't fix it for you? If so this is strange, I can't reproduce the issue with this PR...
As you can see the image made with the master branch is much brighter than the one with this PR. |
Umm.. Sometimes the bottom of the generated image is grayed out; this did not happen when I initialized
I think I have misled you, but there is no doubt that this PR is very good, as you have demonstrated in your examples. |
Hmm that's really weird. I don't see how this could happen.
The result with tiled VAE is always going to be slightly lower quality than the full VAE, because it's lacking information about the whole image. (That's especially true with SD1.x/SD2.x models, because the VAE is quirky ). When using TAE (or with SDXL/SD3 or FLUX), the results are almost completely identical with the PR, while on master, there is the "overexposure" issue on the tiled TAE.
(there is a lot of difference here, mostly because of the kl-f8 vae doesn't handle tiling very well)
(these are almost identical however)
(no tiling) |
|
I can't vouch for the new code being 100% correct, not knowing the code base, but the approach looks reasonable (but does it have to match how the upscaler model was trained, and if so, does it match? I do not know enough for that). But I can definitely vouch for the old code being incorrect. It assumes that tile overlap is equal in x and y direction, and that's unlikely assuming "somewhat arbitrary" total sizes and a fixed tile size. And I can definitely confirm that #666 is fixed by this change - all five test cases pass (up from only the base case), and the more complex case I discovered this with appears to be fixed as well. Now the difference pictures from pre- and post-upscale pictures (after scaling both back to the same size) look like an edge detector without any noticeable artifacts/biases - precisely what's expected. |
d1fcf84 to
fb5f5b7
Compare
|
@stduhpf , would you please rebase this onto master? There are a few conflicts due to the VAE changes for Wan support. |
fb5f5b7 to
c0c0097
Compare
|
I haven't tested it, I know it builds fine, but I'm not sure if it actually works (I can't test it right now) |
c0c0097 to
e78d630
Compare
No problem, I'll test it here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've only noticed a single, very familiar issue 😀
|
Encode is also broken, fixing it |
e78d630 to
d7b671c
Compare
101dcbb to
0a2b4e4
Compare
* refactor tile number calculation * support non-square tiles * add env var to change tile overlap * add safeguards and better error messages for SD_TILE_OVERLAP * add safeguards and include overlapping factor for SD_TILE_SIZE * avoid rounding issues when specifying SD_TILE_SIZE as a factor * lower SD_TILE_OVERLAP limit * zero-init empty output buffer
0a2b4e4 to
987ced8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good, and working fine on a few tests. The only issue I notice was a crash on an invalid tile size (more detailed comments below).
stable-diffusion.h
Outdated
| int tile_size_x; | ||
| int tile_size_y; | ||
| float target_overlap; | ||
| bool relative; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps this boolean could be dropped, and we apply the same logic as the command-line: if rel_size_ is bigger than 0.0, it overrides the tile_size_ ?
stable-diffusion.cpp
Outdated
| if (!use_tiny_autoencoder) { | ||
| float tile_overlap = vae_tiling_params.target_overlap; | ||
| int tile_size_x = vae_tiling_params.tile_size_x; | ||
| int tile_size_y = vae_tiling_params.tile_size_y; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A relative size of 0.0 applies a 4x4 tile (doesn't look good, but works); but a zeroed-out tile size crashes. Maybe an invalid value could just mean the default?
int tile_size_x = vae_tiling_params.tile_size_x < 4 ? 32 : vae_tiling_params.tile_size_x;
int tile_size_y = vae_tiling_params.tile_size_y < 4 ? 32 : vae_tiling_params.tile_size_y;Then we could also simplify the default values for the param struct, by using zeroed-out fields.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same issue with the tile_overlap above.
|
Thanks for the review! I probably won't be able to work on these changes before tomorrow though. |
* avoid crash with invalid tile sizes, use 0 for default * refactor default tile size, limit overlap factor * remove explicit parameter for relative tile size * limit encoding tile to latent size
|
I think this PR can be merged now. Thank you all for your contributions. |
* docs: add sd.cpp-webui as an available frontend (leejet#738) * fix: correct head dim check and L_k padding of flash attention (leejet#736) * fix: convert f64 to f32 and i64 to i32 when loading weights * docs: add LocalAI to README's UIs (leejet#741) * sync: update ggml * sync: update ggml * feat: upgrade musa sdk to rc4.2.0 (leejet#732) * feat: change image dimensions requirement for DiT models (leejet#742) * feat: add missing models and parameters to image metadata (leejet#743) * feat: add new scheduler types, clip skip and vae to image embedded params - If a non default scheduler is set, include it in the 'Sampler' tag in the data embedded into the final image. - If a custom VAE path is set, include the vae name (without path and extension) in embedded image params under a `VAE:` tag. - If a custom Clip skip is set, include that Clip skip value in embedded image params under a `Clip skip:` tag. * feat: add separate diffusion and text models to metadata --------- Co-authored-by: one-lithe-rune <skapusniak@lithe-runes.com> * refector: optimize the usage of tensor_types * feat: support build against system installed GGML library (leejet#749) * chore: avoid setting GGML_MAX_NAME when building against external ggml (leejet#751) An external ggml will most likely have been built with the default GGML_MAX_NAME value (64), which would be inconsistent with the value set by our build (128). That would be an ODR violation, and it could easily cause memory corruption issues due to the different sizeof(struct ggml_tensor) values. For now, when linking against an external ggml, we demand it has been patched with a bigger GGML_MAX_NAME, since we can't check against a value defined only at build time. * Conv2D direct support (leejet#744) * Conv2DDirect for VAE stage * Enable only for Vulkan, reduced duplicated code * Cmake option to use conv2d direct * conv2d direct always on for opencl * conv direct as a flag * fix merge typo * Align conv2d behavior to flash attention's * fix readme * add conv2d direct for controlnet * add conv2d direct for esrgan * clean code, use enable_conv2d_direct/get_all_blocks * format code --------- Co-authored-by: leejet <leejet714@gmail.com> * sync: update ggml, make cuda im2col a little faster * chore: add Nvidia 30 series (cuda arch 86) to build * feat: throttle model loading progress updates (leejet#782) Some terminals have slow display latency, so frequent output during model loading can actually slow down the process. Also, since tensor loading times can vary a lot, the progress display now shows the average across past iterations instead of just the last one. * docs: add missing dash to docs/chroma.md (leejet#771) * docs: add compile option needed by Ninja (leejet#770) * feat: show usage on unknown arg (leejet#767) * fix: typo in the verbose long flag (leejet#783) * feat: add wan2.1/2.2 support (leejet#778) * add wan vae suppport * add wan model support * add umt5 support * add wan2.1 t2i support * make flash attn work with wan * make wan a little faster * add wan2.1 t2v support * add wan gguf support * add offload params to cpu support * add wan2.1 i2v support * crop image before resize * set default fps to 16 * add diff lora support * fix wan2.1 i2v * introduce sd_sample_params_t * add wan2.2 t2v support * add wan2.2 14B i2v support * add wan2.2 ti2v support * add high noise lora support * sync: update ggml submodule url * avoid build failure on linux * avoid build failure * update ggml * update ggml * fix sd_version_is_wan * update ggml, fix cpu im2col_3d * fix ggml_nn_attention_ext mask * add cache support to ggml runner * fix the issue of illegal memory access * unify image loading processing * add wan2.1/2.2 FLF2V support * fix end_image mask * update to latest ggml * add GGUFReader * update docs * feat: add support for timestep boundary based automatic expert routing in Wan MoE (leejet#779) * Wan MoE: Automatic expert routing based on timestep boundary * unify code style and fix some issues --------- Co-authored-by: leejet <leejet714@gmail.com> * feat: add flow shift parameter (for SD3 and Wan) (leejet#780) * Add flow shift parameter (for SD3 and Wan) * unify code style and fix some issues --------- Co-authored-by: leejet <leejet714@gmail.com> * docs: update docs and help message * chore: update to c++17 * docs: update docs/wan.md * fix: add flash attn support check (leejet#803) * feat: support incrementing ref image index (omni-kontext) (leejet#755) * kontext: support ref images indices * lora: support x_embedder * update help message * Support for negative indices * support for OmniControl (offsets at index 0) * c++11 compat * add --increase-ref-index option * simplify the logic and fix some issues * update README.md * remove unused variable --------- Co-authored-by: leejet <leejet714@gmail.com> * feat: add detailed tensor loading time stat (leejet#793) * fix: clarify lora quant support and small fixes (leejet#792) * fix: accept NULL in sd_img_gen_params_t::input_id_images_path (leejet#809) * chore: update flash attention warnings (leejet#805) * fix: use {} for params init instead of memset (leejet#781) * chore: remove sd3 flash attention warn (leejet#812) * feat: use log_printf to print ggml logs (leejet#545) * chore: add install() support in CMakeLists.txt (leejet#540) * feat: add SmoothStep Scheduler (leejet#813) * feat: add sd3 flash attn support (leejet#815) * fix: make tiled VAE reuse the compute buffer (leejet#821) * feat: reduce CLIP memory usage with no embeddings (leejet#768) * fix: make weight override more robust against ggml changes (leejet#760) * fix: do not force VAE type to f32 on SDXL (leejet#716) This seems to be a leftover from the initial SDXL support: it's not enough to avoid NaN issues, and it's not not needed for the fixed sdxl-vae-fp16-fix . * feat: use Euler sampling by default for SD3 and Flux (leejet#753) Thank you for your contribution. * fix: harden for large files (leejet#643) * feat: Add SYCL Dockerfile (leejet#651) * feat: increase work_ctx memory buffer size (leejet#814) * docs: update docs * feat: add VAE encoding tiling support and adaptive overlap (leejet#484) * implement tiling vae encode support * Tiling (vae/upscale): adaptative overlap * Tiling: fix edge case * Tiling: fix crash when less than 2 tiles per dim * remove extra dot * Tiling: fix edge cases for adaptative overlap * tiling: fix edge case * set vae tile size via env var * vae tiling: refactor again, base on smaller buffer for alignment * Use bigger tiles for encode (to match compute buffer size) * Fix edge case when tile is bigger than latent * non-square VAE tiling (#3) * refactor tile number calculation * support non-square tiles * add env var to change tile overlap * add safeguards and better error messages for SD_TILE_OVERLAP * add safeguards and include overlapping factor for SD_TILE_SIZE * avoid rounding issues when specifying SD_TILE_SIZE as a factor * lower SD_TILE_OVERLAP limit * zero-init empty output buffer * Fix decode latent size * fix encode * tile size params instead of env * Tiled vae parameter validation (#6) * avoid crash with invalid tile sizes, use 0 for default * refactor default tile size, limit overlap factor * remove explicit parameter for relative tile size * limit encoding tile to latent size * unify code style and format code * update docs * fix get_tile_sizes in decode_first_stage --------- Co-authored-by: Wagner Bruna <wbruna@users.noreply.github.com> Co-authored-by: leejet <leejet714@gmail.com> * feat: add vace support (leejet#819) * add wan vace t2v support * add --vace-strength option * add vace i2v support * fix the processing of vace_context * add vace v2v support * update docs * feat: optimize tensor loading time (leejet#790) * opt tensor loading * fix build failure * revert the changes * allow the use of n_threads * fix lora loading * optimize lora loading * add mutex * use atomic * fix build * fix potential duplicate issue * avoid duplicate lookup of lora tensor * fix progeress bar * remove unused remove_duplicates --------- Co-authored-by: leejet <leejet714@gmail.com> * refactor: simplify the logic of pm id image loading (leejet#827) * feat: add sgm_uniform scheduler, simple scheduler, and support for NitroFusion (leejet#675) * feat: Add timestep shift and two new schedulers * update readme * fix spaces * format code * simplify SGMUniformSchedule * simplify shifted_timestep logic * avoid conflict --------- Co-authored-by: leejet <leejet714@gmail.com> * refactor: move tiling cacl and debug print into the tiling code branch (leejet#833) * refactor: simplify DPM++ (2S) Ancestral (leejet#667) * chore: set release tag by commit count * chore: fix workflow (leejet#836) * fix: avoid multithreading issues in the model loader * fix: avoid segfault for pix2pix models without reference images (leejet#766) * fix: avoid segfault for pix2pix models with no reference images * fix: default to empty reference on pix2pix models to avoid segfault * use resize instead of reserve * format code --------- Co-authored-by: leejet <leejet714@gmail.com> * refactor: remove unused --normalize-input parameter (leejet#835) * fix: correct tensor deduplication logic (leejet#844) * docs: include Vulkan compatibility for LoRA quants (leejet#845) * docs: HipBLAS / ROCm build instruction fix (leejet#843) * fix: tensor loading thread count (leejet#854) * fix: optimize the handling of CLIP embedding weight (leejet#840) * sync: update ggml * sync: update ggml * fix: optimize the handling of embedding weight (leejet#859) * feat: add support for Flux Controls and Flex.2 (leejet#692) * docs: update README.md (leejet#866) * chore: fix dockerfile libgomp1 dependency + improvements (leejet#852) * fix: ensure directory iteration results are sorted by filename (leejet#858) * chore: fix vulkan ci (leejet#878) * feat: add support for more esrgan models & x2 & x1 models (leejet#855) * feat: add a stand-alone upscale mode (leejet#865) * feat: add a stand-alone upscale mode * fix prompt option check * format code * update README.md --------- Co-authored-by: leejet <leejet714@gmail.com> * refactor: deal with default img-cfg-scale at the library level (leejet#869) * feat: add Qwen Image support (leejet#851) * add qwen tokenizer * add qwen2.5 vl support * mv qwen.hpp -> qwenvl.hpp * add qwen image model * add qwen image t2i pipeline * fix qwen image flash attn * add qwen image i2i pipline * change encoding of vocab_qwen.hpp to utf8 * fix get_first_stage_encoding * apply jeffbolz f32 patch leejet#851 (comment) * fix the issue that occurs when using CUDA with k-quants weights * optimize the handling of the FeedForward precision fix * to_add_out precision fix * update docs * fix: resolve VAE tiling problem in Qwen Image (leejet#873) * fix: avoid generating black images when running T5 on the GPU (leejet#882) * fix: correct canny preprocessor (leejet#861) * fix: better progress display for second-order samplers (leejet#834) * feat: add Qwen Image Edit support (leejet#877) * add ref latent support for qwen image * optimize clip_preprocess and fix get_first_stage_encoding * add qwen2vl vit support * add qwen image edit support * fix qwen image edit pipeline * add mmproj file support * support dynamic number of Qwen image transformer blocks * set prompt_template_encode_start_idx every time * to_add_out precision fix * to_out.0 precision fix * update docs --------- Co-authored-by: Daniele <57776841+daniandtheweb@users.noreply.github.com> Co-authored-by: Erik Scholz <Green-Sky@users.noreply.github.com> Co-authored-by: leejet <leejet714@gmail.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: R0CKSTAR <xiaodong.ye@mthreads.com> Co-authored-by: stduhpf <stephduh@live.fr> Co-authored-by: one-lithe-rune <skapusniak@lithe-runes.com> Co-authored-by: Seas0 <seashkey@gmail.com> Co-authored-by: NekopenDev <197017459+nekopendev@users.noreply.github.com> Co-authored-by: SmallAndSoft <45131567+SmallAndSoft@users.noreply.github.com> Co-authored-by: Markus Hartung <mail@hartmark.se> Co-authored-by: clibdev <52199778+clibdev@users.noreply.github.com> Co-authored-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: rmatif <kingrealriadh@gmail.com> Co-authored-by: vmobilis <75476228+vmobilis@users.noreply.github.com> Co-authored-by: Stefan-Olt <stefan-oltmanns@gmx.net> Co-authored-by: Sharuzzaman Ahmat Raslan <sharuzzaman@gmail.com> Co-authored-by: Serkan Sahin <14278530+SergeantSerk@users.noreply.github.com> Co-authored-by: Pedrito <pedro.c.vfx@gmail.com>














SD_TILE_SIZEenvironment variable. (tile size for vae encoding will be about 1.3x bigger, for the same memory requirement)With these changes, vae encoding can now be tiled, and a small (unreported?) issue with the upscaler (or tae) over-brightening some parts of the image is now fixed. This could also be a step toward tiled diffusion support.
Fixes #588, #564 and #666