Add Timestep shift, SGM Uniform and Simple scheduler and support for NitroFusion

[rmatif]

This PR adds support for the timestep-shift technique required for inference with NitroFusion models and Diff-Instruct* and other one-step models. It also adds support for two schedulers: SGM Uniform and Simple, because the existing scheduler, for a mysterious reason, fails at step 2 calculation and produces an output similar to step 1.

NitroFusion is one of the best models for single-step inference, making it useful for inference on compute-constrained devices like mobile phones or CPUs.

Example command:

./bin/sd -m nitrosd-realism_f16.gguf -v -p "cute cat" --cfg-scale 1 --steps 1 --timestep-shift 250 -H 1024 -W 1024 --seed 2024 --schedule sgm_uniform

Step	NitroSD-Realism (Timestep-shift 250)	NitroSD-Vibrant (Timestep-shift 500)	Diff-Instruct* (Timestep-shift 400)
Step 1
Step 2			Not well supported
Step 3			Not well supported
Step 4			Not well supported

The recommended timestep-shift values by the authors are 250 for NitroSD-Realism and 500 for NitroSD-Vibrant and 400 for Diff-Instruct*.

I created GGUF versions of NitroFusion that already include the fixed SDXL VAE, available for download here.

The authors mentioned it's possible to extract LoRA weights from these models and apply them to other checkpoints. I’ll try to do that in the future.

EDIT : Just add Diff-Instruct* GGUF
References:

Timestep-shift implementation: node.py
SGM Uniform: sd_schedulers
Simple: samplers.py

[Green-Sky]

Tested simple in isolation, and it works 👍

[Green-Sky]

trailing space

[Green-Sky]

Please rebase and update this pr on master. It is not nice to test right now.
Also, to aide in mergability, please split the individual features, when possible. (like the simple schedule)

I am currently trying to get sgm to work. :)

[Green-Sky]

fixes #198

[rmatif]

Please rebase and update this pr on master. It is not nice to test right now. Also, to aide in mergability, please split the individual features, when possible. (like the simple schedule)

I am currently trying to get sgm to work. :)

I'll try to do it next week. I think the changes are small enough to be integrated into a single PR, but if that's not preferred, I can make a separate PR for the new schedulers and the timestep shift

@Green-Sky Rebase complete. Can you test it now? hopefully I didn’t break anything

[Green-Sky]

Can confirm SGM Uniform works too.

[Green-Sky]

I saw some model authors recommend the simple schedule for flux models, so I tried it.

Using my custom hyper-flux.1-lite-8b-8step q5_k

discrete	simple

Honestly hard to say which one is better, 1 and 3 look better simple to me, but 2 looks better discrete. 🤷

[rmatif]

Honestly hard to say which one is better, 1 and 3 look better simple to me, but 2 looks better discrete. 🤷

Guess it's a matter of taste. I read somewhere a while ago that SGM Uniform performs better with distilled models. Now it's up to @leejet to merge it if everything looks good for him

[Green-Sky]

@leejet the simple scheduler is pretty common now, and required for chroma-flash (+heun), so would really love to see this merged.

@rmatif there where conflicting changes in master, please update the pr with resolutions :)

[rmatif]

@rmatif there where conflicting changes in master, please update the pr with resolutions :)

Sorry, I don’t have much time these days. If @leejet wants to take over and resolve the conflict, that would be great. Otherwise, I’ll try to get back to it when I have some time

[leejet]

@rmatif there where conflicting changes in master, please update the pr with resolutions :)

Sorry, I don’t have much time these days. If @leejet wants to take over and resolve the conflict, that would be great. Otherwise, I’ll try to get back to it when I have some time

@rmatif Thank you for your contribution. I'll find some time to resolve the conflicts and merge this PR.

[wbruna]

@leejet , if I may make a suggestion:

        // Check if the current schedule is SGMUniformSchedule
        if (std::dynamic_pointer_cast<SGMUniformSchedule>(schedule)) {
            std::vector<float> sigs;

This isn't the first time the current class hierarchy and code organization gets in the way of scheduler's algorithms: DDIM and TCD have to undo parts of StableDiffusionGGML::sample to work, and that mismatch also causes bugs like #663 . Maybe these mismatches could be avoided by a different class hierarchy?

[leejet]

@leejet , if I may make a suggestion:

        // Check if the current schedule is SGMUniformSchedule
        if (std::dynamic_pointer_cast<SGMUniformSchedule>(schedule)) {
            std::vector<float> sigs;

This isn't the first time the current class hierarchy and code organization gets in the way of scheduler's algorithms: DDIM and TCD have to undo parts of StableDiffusionGGML::sample to work, and that mismatch also causes bugs like #663 . Maybe these mismatches could be avoided by a different class hierarchy?

@wbruna Actually, the sampling-related code does need to be refactored, and I’m planning to do that soon. But in this case, it’s not strictly required by the code structure—I’ve simplified the relevant logic instead, so feel free to take a look.

[wbruna]

Testing 58225a0 , single-step gens:

Nitro Vibrant (500)	Nitro Realism (250)	Diff-Instruct* (400)

[leejet]

It looks like this PR can be merged now. Thank you everyone!