DDCOT

[NeurIPS 2023]DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models This repository is the official implementation of DDCoT.

DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

Ge Zheng∗ , Bin Yang∗, Jiajin Tang*, Hong-Yu Zhou, Sibei Yang†

*Equal contribution; †Corresponding Author

Requirements

Install all required python dependencies:

pip install -r requirements.txt

Datasets

Download the dataset from the following repository:

https://github.com/lupantech/ScienceQA/tree/main/data

Instructions

Our trained model, extracted visual features and captions and generated rationales are available at Google Drive

Generate Rationale

To generate rationale for one sample，you can run the following code.

python rationale_generation.py

Train

python main.py \
    --model  allenai/unifiedqa-t5-base \
    --user_msg answer --img_type clip \
    --bs 16 --eval_bs 4 --eval_acc 10 --output_len 64 --lr 1e-4 \
    --prompt_format QCMG-A \
    --output_dir model_path --use_generate --final_eval

Inference

python main.py \
    --model allenai/unifiedqa-t5-base \
    --user_msg answer --img_type clip \
    --bs 8 --eval_bs 1 --eval_acc 10 --output_len 64 \
    --final_eval --prompt_format QCMG-A \
    --evaluate_dir model_path --use_generate 

Citing DDCoT

@article{zheng2023ddcot,
  title={Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models},
  author={Zheng, Ge and Yang, Bin and Tang, Jiajin and Zhou, Hong-Yu and Yang, Sibei},
  journal={arXiv preprint arXiv:2310.16436},
  year={2023}
}

License

This project is licensed under the Apache-2.0 License.

Acknowledgement

Part of our codes are adapted from ScienceQA.