Skip to content

Latest commit

 

History

History
280 lines (239 loc) · 10.3 KB

TRAIN_AND_VALIDATE.md

File metadata and controls

280 lines (239 loc) · 10.3 KB
Raw

Data preparation

data for training

We provide the processed data on Hugging Face, or you also can download from Baidu Disk as follows.

DatasetsBaidu Disk
Image pretrainingLink
Image tuningLink
Video pretrainingLink
Video tuningLink

After downloading all of them, organize the data as follows in DATA_ROOT.

DATA_ROOT
├── llava_image
├── llava_image_tune
├── valley
└── videochatgpt_tune

data for validating

  • For image, follow LLaVA's instructions. You MUST first download eval.zip. It contains custom annotations, scripts, and the prediction files with LLaVA v1.5. Extract to eval. This also provides a general structure for all datasets.
  • For video, videos and annotations can be downloaded from Video-ChatGPT. We also provide the processed data as follows.
DatasetsBaidu DiskGoogle DiskPeking University Disk
Activitynet_Zero_Shot_QALink--
MSRVTT_Zero_Shot_QALinkLink-
MSVD_Zero_Shot_QALinkLinkLink
TGIF_Zero_Shot_QALinkLinkLink

After downloading all of them, organize the data as follows in eval.

eval
├── GPT_Zero_Shot_QA
│   ├── Activitynet_Zero_Shot_QA
│   ├── MSRVTT_Zero_Shot_QA
│   ├── MSVD_Zero_Shot_QA
│   └── TGIF_Zero_Shot_QA
├── gqa
│   ├── answers
│   ├── data
│   └── llava_gqa_testdev_balanced.jsonl
├── llava-bench-in-the-wild
│   ├── answers
│   ├── answers_gpt4.jsonl
│   ├── bard_0718.jsonl
│   ├── bing_chat_0629.jsonl
│   ├── context.jsonl
│   ├── images
│   ├── questions.jsonl
│   ├── README.md
│   └── reviews
├── mmbench
│   ├── answers
│   ├── answers_upload
│   ├── mmbench_dev_20230712.tsv
│   └── mmbench_dev_en_20231003.tsv
├── MME
│   ├── answers
│   ├── convert_answer_to_mme.py
│   └── llava_mme.jsonl
├── mm-vet
│   ├── answers
│   ├── bard_set.json
│   ├── convert_answers.py
│   ├── images
│   ├── llava-mm-vet.jsonl
│   ├── mm-vet.json
│   └── results
├── pope
│   ├── answers
│   ├── coco
│   ├── llava_pope_test.jsonl
│   └── val2014
├── scienceqa
│   ├── answers
│   ├── images
│   ├── llava_test_CQM-A.json
│   ├── pid_splits.json
│   └── problems.json
├── seed_bench
│   ├── answers
│   ├── answers_upload
│   ├── extract_video_frames.py
│   └── llava-seed-bench.jsonl
├── textvqa
│   ├── answers
│   ├── llava_textvqa_val_v051_ocr.jsonl
│   ├── TextVQA_0.5.1_val.json
│   └── train_images
├── vizwiz
│   ├── answers
│   ├── answers_upload
│   ├── llava_test.jsonl
│   ├── test
│   ├── test.json
│   ├── train.json
│   └── val.json
└── vqav2
    ├── answers
    ├── answers_upload
    ├── llava_vqav2_mscoco_test2015.jsonl
    ├── llava_vqav2_mscoco_test-dev2015.jsonl
    └── test2015

Training

Specify your DATA_ROOT according to the data preparation.

Validating

Our image validation code comes from LLaVA and our video validation code comes from Video-ChatGPT, thanks for their contribution!

You can refer to the official repository for validation, but we also provide off-the-shelf scripts.

To load unmerged LoRA weights, you simply need to pass an additional argument --model-base, which is the base LLM that is used to train the LoRA weights.

MSRVTT-QA

  1. Inference to get the result.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/run_qa_msrvtt.sh
  1. GPT-Assistant evaluation.
bash scripts/v1_5/eval/eval_qa_msrvtt.sh

MSVD-QA

  1. Inference to get the result.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/run_qa_msvd.sh
  1. GPT-Assistant evaluation.
bash scripts/v1_5/eval/eval_qa_msvd.sh

TGIF-QA

  1. Inference to get the result.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/run_qa_tgif.sh
  1. GPT-Assistant evaluation.
bash scripts/v1_5/eval/eval_qa_tgif.sh

ActivityNet-QA

  1. Inference to get the result.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/run_qa_activitynet.sh
  1. GPT-Assistant evaluation.
bash scripts/v1_5/eval/eval_qa_activitynet.sh

VQAv2

  1. Download test2015 and put it under eval/vqav2.
  2. Multi-GPU inference.
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/eval_image_vqav2.sh
  1. Submit the results to the evaluation server: eval/vqav2/answers_upload.

GQA

  1. Download the data following the official instructions here and put under eval/gqa/data.
  2. Multi-GPU inference.
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/eval_image_gqa.sh

VisWiz

  1. Download test.json and extract test.zip to test. Put them under eval/vizwiz.
  2. Single-GPU inference.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_vizwiz.sh
  1. Submit the results to the evaluation server: eval/vizwiz/answers_upload.

ScienceQA

  1. Under eval/scienceqa, download images, pid_splits.json, problems.json from the data/scienceqa folder of the ScienceQA repo.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_sqa.sh

TextVQA

  1. Download TextVQA_0.5.1_val.json and images and extract to eval/textvqa.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_textvqa.sh

POPE

  1. Download coco from POPE and put under eval/pope.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_pope.sh

MMBench

  1. Download mmbench_dev_20230712.tsv and put under eval/mmbench.
  2. Single-GPU inference.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_mmbench.sh
  1. Submit the results to the evaluation server: eval/mmbench/answers_upload/mmbench_dev_20230712.

LLaVA-Bench-in-the-Wild

  1. Extract contents of llava-bench-in-the-wild to eval/llava-bench-in-the-wild.
  2. Single-GPU inference and evaluate.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_llavabench.sh

MM-Vet

  1. Extract mm-vet.zip to eval/mmvet.
  2. Single-GPU inference.
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/eval_image_mmvet.sh