Skip to content

lumina reversed timesteps seems not handled in some funcs #2201

@urlesistiana

Description

@urlesistiana

During my training, I accidently found if I use the default shift timestep sampling method, even after thousands of steps, the model still learned nothing, the sample images are identical. Which is very strange. Because when using nextdit_shift it only needs ~200 steps to show the noticeable changes in sample images

Lumina uses t=0 as the noise and t=1 as the image. Which is reversed, unlike other models (t=1 as the noise and t=0 as image).

I found this causes some funcs not working as expected, since they expects the usual timesteps.

In

def get_noisy_model_input_and_timesteps(

nextdit_shift reversed the timesteps in time_shift(). Other timestep sampling methods don't work properly. They still shift toward t=1.

--min_snr_gamma is also affected, maybe more.

I also found that diffusers still uses the usual timesteps in its api, and only reverse it before passing to the network.

https://github.com/huggingface/diffusers/blob/0f252be0ed42006c125ef4429156cb13ae6c1d60/src/diffusers/pipelines/lumina2/pipeline_lumina2.py#L698

Maybe we should do this as well?

t=timesteps / 1000, # timesteps需要除以1000来匹配模型预期

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions