Pipeline: obtain per-image reproducibility seeds when generating multiple images

[pcuenca]

People are interested in the following workflow: generate a few images from a prompt, select the one you like and tweak the prompt to steer it towards your goal. There are many examples in Twitter and the Discord bot, see context section below.

I managed to replicate this workflow in a demo Space generating one image at a time, and manually setting the seed before each generation. This is sub-optimal, especially when considering high-load services that need to batch requests from several users in order to run them in parallel.

Given that the pipeline is already designed to receive a Generator, I'd like to explore if we can (optionally) return per-image seeds as part of the generation process.

Describe the solution you'd like
I'd like the following to work:

ldm = DiffusionPipeline.from_pretrained(MODEL_ID)
rng = torch.manual_seed(1337)
prompts = ["Labrador in the style of Vermeer"] * 6
preds = ldm(prompts, generator=rng, return_seeds=True)
images, seeds = preds["sample"], preds["seed"]

rng.manual_seed(seeds[3])
tweaked_image = ldm("Labrador in the style of Hokusai", generator=rng)

The same thing should work when batching prompts from several users.

Perhaps something like this is already possible and I haven't looked enough.

One idea is to iterate here instead of creating the latents all at once. It would have a performance penalty, but I presume it should be negligible compared to the rest of the process. A drawback of this approach is that we'd need to do it for every pipeline, or try to create a helper.

Describe alternatives you've considered

• Do nothing, and use the low-level API instead of the pipeline to achieve this goal (I think it should be possible, I plan to test it later). Implementing it in the pipeline would just make it easier for some users to experiment on their own.
• Iterate to obtain just a single image per generation. This wastes GPU memory and is slower in most cases. But it might not be a problem for users that just want to experiment.

Additional context

• This request
• Discord:
  
  (screenshot credit: @osanseviero)

[anton-l]

This would be a cool feature indeed!
Since torch.Generator() operates with ByteTensor get_state() and set_state(), for this we might need to replace the pipe(generator) argument with just an integer seed, like we've discussed earlier with @patil-suraj @patrickvonplaten

[spychiatrist]

Coming here from the HuggingFace comments. I've implemented a simple workaround for the non-determinism: sequential seeding (for n_iter) now with sub-seed offset (for n_samples). One new parameter --seed_offset allows you to deterministically shift which sample-subseed is selected.

Here's an example of deterministically shifting down and right in the matrix (previously impossible):

	--n_samples 3 --n_iter 3 --seed 42 --seed_offset 0	--n_samples 2 --n_iter 2 --seed 43 --seed_offset 1

See this commit (and priors) on my fork of the main repo

Other relevant commit w/ simple sequential seeding

[atarashansky]

@spychiatrist Not a hugging-face dev, but would love to see a PR for this in the base repo.

[patrickvonplaten]

Coming here from the HuggingFace comments. I've implemented a simple workaround for the non-determinism: sequential seeding (for n_iter) now with sub-seed offset (for n_samples). One new parameter --seed_offset allows you to deterministically shift which sample-subseed is selected.

Here's an example of deterministically shifting down and right in the matrix (previously impossible):

--n_samples 3 --n_iter 3 --seed 42 --seed_offset 0 --n_samples 2 --n_iter 2 --seed 43 --seed_offset 1

See this commit (and priors) on my fork of the main repo

Other relevant commit w/ simple sequential seeding

That's a smart idea with seed_offset @spychiatrist !
I see the need and the problem, also in favor of finding a good solution here!

To narrow down the problem here a bit more:

If I take a seed=0 (or a generator state, IMO it doesn't really matter) and I generate gaussian noise of size (4,3), then there is IMO no way to generate gaussian noise of size (1,3) afterwards that corresponds to the first batch_idx of (4,3) except ones generates the tensor of the previous batch size which is API-wise not feasible.

I don't really see how this can be solved if we continue to create images via:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Line 85 in e30e1b8

latents = torch.randn(

(e.g. generating random noise for the whole batch)
=> we're closer to a solution if we do something like:

        latents = torch.cat([torch.randn(
            (1, self.unet.in_channels, height // 8, width // 8),
            generator=generator,
            device=torch_device,
        ) for _ in range(batch_size)], dim=0)

I think, because than we always create the same tensor sizes at least.
In that case, it would also indeed be nicer if we would use seeds (maybe with offsets) or we let the user pass seeds as suggested by @anton-l

One other way that would put a higher burden on the user, but would very much help us in terms of API and managing complexity would be to maybe just introduce a latents: Optional[torch.Tensor] = None argument to the __call__ function so that the user can pass the noise themselves. This way no random seed or generator is needed at all because the pipeline is deterministic (think that's my favorite approach for now).

4. Adding seed in addition to generator I think is a bit weird. We could replace generator with seed though if we notice seed has more advantages! Also cc @LysandreJik and @sgugger regarding seed vs. generator in PyTorch since we talked about it a bit.

[patrickvonplaten]

@anton-l @patil-suraj what do you think here? IMO, let's just add latents as an argument to the pipeline no?

[pcuenca]

@anton-l @patil-suraj what do you think here? IMO, let's just add latents as an argument to the pipeline no?

I'm exploring something else in a custom pipeline: pass an array of seeds that defaults to None. I can submit a PR for discussion.

[leszekhanusz]

So, using the same seed using either diffusers or directly the stable-diffusion library will generate different pictures?

[pcuenca]

Fixed in #247.