First Commit

focus000 · Mar 9, 2023 · 5a565dd · 5a565dd
commit 5a565dd
Show file tree

Hide file tree

Showing 13 changed files with 1,119 additions and 0 deletions.
diff --git a/.idea/.gitignore b/.idea/.gitignore
diff --git a/.idea/inspectionProfiles/Project_Default.xml b/.idea/inspectionProfiles/Project_Default.xml
diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml
diff --git a/.idea/misc.xml b/.idea/misc.xml
diff --git a/.idea/modules.xml b/.idea/modules.xml
diff --git a/.idea/vcs.xml b/.idea/vcs.xml
diff --git a/.idea/visual-chatgpt.iml b/.idea/visual-chatgpt.iml
diff --git a/README.md b/README.md
@@ -0,0 +1,46 @@
+# Visual ChatGPT 
+
+**Visual ChatGPT** connects ChatGPT and a series of Visual Foundation Models to enable **sending** and **receiving** images during chatting.
+
+See our paper: [<font size=5>Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models</font>](https://arxiv.org/abs/2303.04671)
+
+## Demo 
+<img src="./assets/demo.gif" width="750">
+
+##  System Architecture 
+
+
+<p align="center"><img src="./assets/figure.jpg" alt="Logo"></p>
+
+
+## Quick Start
+
+```
+# create a new environment
+conda create -n visgpt python=3.8
+
+#  prepare the basic environments
+pip install -r requirement.txt
+
+# download the visual foundation models
+bash download.sh
+
+# prepare your private openAI private key
+export OPENAI_API_KEY={Your_Private_Openai_Key}
+
+# crate a folder to save images
+mkdir ./image
+
+# Start Visual ChatGPT !
+python visual_chatgpt.py
+```
+
+
+## Acknowledgement
+We appreciate the open source of the following projects:
+
+- HuggingFace [[Project]](https://github.com/huggingface/transformers)
+
+- ControlNet  [[Paper]](https://arxiv.org/abs/2302.05543) [[Project]](https://github.com/lllyasviel/ControlNet)
+
+- Stable Diffusion [[Paper]](https://arxiv.org/abs/2112.10752)  [[Project]](https://github.com/CompVis/stable-diffusion)
diff --git a/assets/demo.gif b/assets/demo.gif
diff --git a/assets/figure.jpg b/assets/figure.jpg
diff --git a/download.sh b/download.sh
@@ -0,0 +1,13 @@
+git clone https://github.com/lllyasviel/ControlNet.git
+ln -s ControlNet/ldm ./ldm
+ln -s ControlNet/cldm ./cldm
+ln -s ControlNet/annotator ./annotator
+cd ControlNet/models
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_canny.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_depth.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_hed.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_mlsd.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_normal.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_openpose.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_scribble.pth
+wget https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_seg.pth
diff --git a/requirement.txt b/requirement.txt
@@ -0,0 +1,30 @@
+torch==1.12.1
+torchvision==0.13.1
+numpy==1.23.1
+transformers==4.26.1
+albumentations==1.3.0
+opencv-contrib-python==4.3.0.36
+imageio==2.9.0
+imageio-ffmpeg==0.4.2
+pytorch-lightning==1.5.0
+omegaconf==2.1.1
+test-tube>=0.7.5
+streamlit==1.12.1
+einops==0.3.0
+webdataset==0.2.5
+kornia==0.6
+open_clip_torch==2.0.2
+invisible-watermark>=0.1.5
+streamlit-drawable-canvas==0.8.0
+torchmetrics==0.6.0
+timm==0.6.12
+addict==2.4.0
+yapf==0.32.0
+prettytable==3.6.0
+safetensors==0.2.7
+basicsr==1.4.2
+langchain==0.0.101
+diffusers
+gradio
+openai
+accelerate