I can't generate meaningful results on CIFAR10

[huanranchen]

Hi~
Your work inspire me a lot and I'm really appreciate for such meaningful and excellent work. However I fail to generate meaningful images on cifar10.. Could you help me find out what's wrong in my code?

I think it's might be easy to generate images without classifier's gradient, so I first try to generate images only by the gradient from diffusion model. Here is my code:

            x = torch.randn(1, *img_shape, device=self.device)
            x = x * 0.5 + 0.5
            x.requires_grad = True
            optimizer = torch.optim.Adam([x], lr=2e-2)
            for _ in tqdm(range(iter_each_sample)):  # iterate for 1000 times
                optimizer.zero_grad()
                print(self.partial(x, class_id))  # get the gradient of 
                optimizer.step()
            x.grad = None
            x.requires_grad = False
            x = torch.clamp(x, min=0, max=1)

Here, class_id is None, and self.partial is a function that get the gradient of :

I use NCSNpp architecture and model weight from Yang Song's official code.

But finally, images generated by this code is like this:

It's not meaningful... Could you help find what's wrong here?

[AlexGraikos]

Hello,

Can you elaborate on how iter_each_sample works? We show in the paper that instead of optimizing for all timesteps simultaneously, we anneal t according to some pre-defined schedule (Figure B.1 in the appendix for instance).

If you are not doing this already, I suggest trying it out first to see if the generated samples make a bit more sense.

[huanranchen]

Hello,

Can you elaborate on how iter_each_sample works? We show in the paper that instead of optimizing for all timesteps simultaneously, we anneal t according to some pre-defined schedule (Figure B.1 in the appendix for instance).

If you are not doing this already, I suggest trying it out first to see if the generated samples make a bit more sense.

Thank you~ It seems to be more meaningful if we use annealing schedule for t and not optimizing all t simultaneously. But I have a question, why optimizing for all timesteps simultaneously leads to not meaningful results? According to the derivation in your paper, we should optimizing for all timesteps simultaneously instead of only optimizing one timestep....

[AlexGraikos]

Are you sure you are optimizing for all 1000 timesteps simultaneously? From the derivation in the paper, you actually need to sum the denoising terms and backpropagate through all T of them. Unless you have a very small denoiser network, or you do some accumulation trick, this is hard to perform.

[huanranchen]

Are you sure you are optimizing for all 1000 timesteps simultaneously? From the derivation in the paper, you actually need to sum the denoising terms and backpropagate through all T of them. Unless you have a very small denoiser network, or you do some accumulation trick, this is hard to perform.

Hi~ I'm sure, and my architecture is ncsnpp, I use accumulation trick, which backward 64 timesteps at each time and accumulate 1000/64<16 times...
So it takes me 1 hour to synthesize one image....

[huanranchen]

Are you sure you are optimizing for all 1000 timesteps simultaneously? From the derivation in the paper, you actually need to sum the denoising terms and backpropagate through all T of them. Unless you have a very small denoiser network, or you do some accumulation trick, this is hard to perform.

Could I ask a question, why optimizing for all timesteps simultaneously leads to not meaningful results on CIFAR10? According to the derivation in your paper, we should optimizing for all timesteps simultaneously instead of only optimizing one timestep.......

[ByChelsea]

Are you sure you are optimizing for all 1000 timesteps simultaneously? From the derivation in the paper, you actually need to sum the denoising terms and backpropagate through all T of them. Unless you have a very small denoiser network, or you do some accumulation trick, this is hard to perform.

Could I ask a question, why optimizing for all timesteps simultaneously leads to not meaningful results on CIFAR10? According to the derivation in your paper, we should optimizing for all timesteps simultaneously instead of only optimizing one timestep.......

I'm also very confused about this. But you mentioned that you backward 64 timesteps at each time, I think "simultaneously" might mean 1000 timesteps.

And I have another question. I cannot understand that "From the derivation in the paper, you actually need to sum the denoising terms and backpropagate through all T of them." I also tried random t (backward 1 timestep each time), it doesn't work. I wonder why this happens...