16k VQGAN Mannequins

[afiaka87]

#319

https://www.dropbox.com/s/x9gbxiartrqaewf/afiaka_544I_96TSL_12D_16H_64DH_sparse.zip

Colab notebook for inference!
https://colab.research.google.com/drive/11V2xw1eLPfZvzW8UQyTUhqCEU71w6Pr4?usp=sharing

Just reposting this here to close the issue

[afiaka87]

@afiaka87 Works great!
a female mannequin dressed in a bright blue jacket and bright red skirt

[afiaka87]

a female mannequin dressed in an olive button-down shirt and gold palazzo pants

an emoji of a baby penguin wearing a blue hat, blue gloves, red shirt, and green pants

a photo of westwood park, san francisco, from the water in the afternoon

a male mannequin dressed in an olive and black checkered button-down shirt and black pleated trousers

a photo of the city streets of france

[soon-yau]

@afiaka87 I'm interested in training a model to generate mannequins, can you share the link of the dataset?

[afiaka87]

@soon-yau no problem - let me find it really quick.

[afiaka87]

@soon-yau
Here's a link to the images:
https://www.dropbox.com/s/a4jx0pe6oc1e5r7/openai_dalle_gen.tar.gz

and here are the captions:
https://www.dropbox.com/s/p0qwhefid4p8q0u/blog_captions.tar.gz

This contains the roughly 200K captioned pairs of mannequins wearing clothes, as well as around 800K captioned pairs for a variety of other situations.

mkdir Dataset
cd Dataset
wget --continue https://www.dropbox.com/s/p0qwhefid4p8q0u/blog_captions.tar.gz
wget --continue https://www.dropbox.com/s/a4jx0pe6oc1e5r7/openai_dalle_gen.tar.gz

tar xvf blog_captions.tar.gz
tar xvf openai_dalle_gen.tar.gz

[soon-yau]

Thanks @afiaka87 !!!!!!

[wangcong9299]

I am trying to download the link you shared, but it's no longer valid, can you share the link of the dataset again? Thank you very much.

[wangcong9299]

@afiaka87

[wangcong9299]

I am trying to download the link you shared, but it's no longer valid, can you share the link of the dataset again? Thank you very much.

[afiaka87]

Sure, let me find it real quick.

[soon-yau]

@afiaka87 Do you know the origin of the dataset e.g. dataset name, authors, paper, university etc? I'd like to cite that in a paper that I'm planning to write.

[afiaka87]

@wangcong9299

These are the images:
https://www.dropbox.com/s/a4jx0pe6oc1e5r7/openai_dalle_gen.tar.gz

Captions:
https://www.dropbox.com/s/p0qwhefid4p8q0u/blog_captions.tar.gz?dl=0

To pair the captions with images, you will want to use the stem (e.g.<stem>.png or <stem>.txt) of the filename they share. Sorry they are a bit long.

[afiaka87]

@soon-yau The original images are all available from OpenAI's CDN on this specific blog post https://openai.com/blog/dall-e/. To scrape the full set of image caption pairs as I've done, you have to dig into the websites JavaScript. My understanding is that images provided from their CDN are licensed under creative commons, although you may want to double check this if it is relevant.

As for the authors, they are the authors of the DALLE paper and blog post. They can be found at https://openai.com/blog/dall-e/ (at the bottom).

[peipeichenml]

Hi @afiaka87 I am trying to run your colab notebook, but I get this error:

Traceback (most recent call last):
  File "/content/dalle-pytorch-pretrained/DALLE-pytorch/generate.py", line 106, in <module>
    dalle.load_state_dict(weights)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1498, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DALLE:
	Missing key(s) in state_dict: "transformer.pos_emb", "transformer.layers.layers.0.0.fn.fn.fn.fn.fn.to_qkv.weight", "transformer.layers.layers.0.0.fn.fn.fn.fn.fn.to_out.0.weight", "transformer.layers.layers.0.0.fn.fn.fn.fn.fn.to_out.0.bias", "transformer.layers.layers.0.0.fn.fn.fn.fn.fn.attn_fn.master_layout", "transformer.layers.layers.0.1.fn.fn.fn.fn.net.0.weight", "transformer.layers.layers.0.1.fn.fn.fn.fn.net.0.bias", "transformer.layers.layers.0.1.fn.fn.fn.fn.net.3.weight", ...

Do you probably know what is wrong?
Thank you!

[afiaka87]

Not sure, this codebase and checkpoint are very old. I've moved on to GLIDE and Latent Diffusion, recommend the same.

[peipeichenml]

Thanks for replying. I will take a look at your recommendation :)