updated experiments

sMamooler · Jan 6, 2022 · c90578d · c90578d
1 parent ad105cf
commit c90578d
Show file tree

Hide file tree

Showing 241 changed files with 1,477 additions and 3,282 deletions.
diff --git a/Images/artemis.jpg b/Images/artemis.jpg
diff --git a/Images/astronaut.png b/Images/astronaut.png
diff --git a/Images/el2.jpeg b/Images/el2.jpeg
diff --git a/Images/el4.jpeg b/Images/el4.jpeg
diff --git a/Images/elephant.jpg b/Images/elephant.jpg
diff --git a/Images/zebra.jpeg b/Images/zebra.jpeg
diff --git a/Images/zebra.jpg b/Images/zebra.jpg
diff --git a/README.md b/README.md
@@ -1 +1,44 @@
-# sem_project_IVRL
+# CLIP Explainability
+
+This repo contains the code for the [CLIP Explainability project](CLIP_Explainability.pdf). 
+In this project, we conduct an in-depth study of CLIP’s learned image and text representations using saliency map visualization. We propose a modification to the existing saliency visualization method that improves its performance as shown by our qualitative evaluations. We then use this method to study CLIP’s ability in capturing similarities and dissimilarities between an input image and targets belonging to different domains including image, text, and emotion.
+
+## Setup
+
+To install the required libraries run the following command:
+
+```
+pip install -r requirements.txt
+
+```
+
+## Organzation
+
+[code](code) directory contains 
+
+- the implementation of saliency visualization methods: for [ViT] (code/vit_cam.py) and ResNet (code/rn_cam.py)-based CLIP
+- Gildenblat et al.'s implementation of [GradCAM](code/pytorch-grad-cam)
+- A re-implementation of CLIP taken from [Transformer-MM-Explainability](https://github.com/hila-chefer/Transformer-MM-Explainability) repo that keeps tack of attention maps and gradients: [clip_.py](code/clip_.py)
+- [Notebooks](code/notebooks/) for the experiments explained in the report
+
+
+[Images](Images) contains images used in the experiments. 
+
+[results](results) contains the results obtained from the experiments. Any result generated by the notebooks will be stored in this directory.
+
+
+## Experiments
+
+
+| Notebook Name |   Experiment  |      Note     |
+| ------------- | ------------- | ------------- |
+| [vit_block_vis](code/notebook/vit_block_vis.ipynb)  | Layer-wise Attention Visualization  |   -    |
+| [saliency_method_compare](code/notebook/saliency_method_compare.ipynb)  | ViT Explainability Method Comparison |  qualitative comparison |
+| [affectnet_emotions](code/notebook/affectnet_emotions.ipynb)  | ViT Explainability Method Comparison |  bias comparison. you need to download a sample of the AffectNet dataset [here](https://drive.google.com/drive/u/1/folders/11RusPab71wGw6LTd9pUnY1Gz3JSH-N_N) and place it in [Images](Images). |
+| [pos_neg_vis](code/notebook/pos_neg_vis.ipynb)  | Positive vs Negative Saliency | - |  
+| [artemis_emotions](code/notebook/artemis_emotions.ipynb)  |  Emotion-Image Similarity  | you need to download the pre-processed WikiArt images [here](https://drive.google.com/drive/u/1/folders/11RusPab71wGw6LTd9pUnY1Gz3JSH-N_N) and place it in [Images](Images). Note that this notebook chooses images randomly so the results may not be the same as the ones in the report. |
+| [perword_vis](code/notebook/perword_vis.ipynb)  | Word-Wise Saliency Visualization  | 
+| [global_vis](code/notebook/global_vis.ipynb)  | - | can be used to visualize saliency maps for ViT and ResNet-based CLIP.
+ can be used to reporoduce the results for bias detection.|
+
+
diff --git a/code/README.md b/code/README.md
diff --git a/code/notebooks/affectnet_emotions.ipynb b/code/notebooks/affectnet_emotions.ipynb
diff --git a/code/notebooks/artemis_emotions.ipynb b/code/notebooks/artemis_emotions.ipynb
diff --git a/code/notebooks/cross_image_saliency.ipynb b/code/notebooks/cross_image_saliency.ipynb
diff --git a/code/notebooks/global_vis.ipynb b/code/notebooks/global_vis.ipynb
diff --git a/code/notebooks/perword_vis.ipynb b/code/notebooks/perword_vis.ipynb
diff --git a/code/notebooks/pos_neg_vis.ipynb b/code/notebooks/pos_neg_vis.ipynb