Chat-UniVi-13B.md

Chat-UniVi-13B

We releas Chat-UniVi-13B. Our proposed unified visual representation framework greatly reduces the number of visual tokens, so you can train 13B unified image and video understanding models in full parameters directly on 8 A100 GPUs within 3 days.

Main Results

Image understanding

Methods	LLM	Conversation	Detail Description	Complex Reasoning	All
Chat-UniVi-7B	Vicuna-7B	84.1	74.2	93.7	84.2
Chat-UniVi-13B	Vicuna-13B	84.1	79.4	94.7	86.1

Video understanding

Methods	LLM	Correct	Detail	Context	Temporal	Consistency
Chat-UniVi-7B	Vicuna-7B	57.8	58.2	69.2	57.8	56.2
Chat-UniVi-13B	Vicuna-13B	59.4	59.8	70.5	58.0	60.6

ScienceQA

Methods	LLM	Average	Subject			Context Modality			Grade
			NAT	SOC	LAN	TXT	IMG	NO	G1-6	G7-12
Chat-UniVi-7B	Vicuna-7B	88.78	88.50	93.03	85.91	88.51	85.97	88.15	88.88	88.60
Chat-UniVi-13B	Vicuna-13B	90.99	90.41	95.05	88.91	89.64	88.05	90.94	91.19	90.64

VideoQA

Methods	LLM	MSRVTT-QA		MSVD-QA		TGIF-QA		ActivityNet-QA
		Accuracy	Score	Accuracy	Score	Accuracy	Score	Accuracy	Score
Chat-UniVi-7B	Vicuna-7B	54.6	3.1	65.0	3.6	60.3	3.4	45.8	3.2
Chat-UniVi-13B	Vicuna-13B	-	-	-	-	-	-	46.4	3.6

Hallucination Evaluation (POPE)

Coming soon.

Train the 13B model

All you have to do is replace the base model with the 13B model.

Stage1: Multimodal Pre-training

deepspeed \
--include localhost:0,1,2,3,4,5,6,7 \
--master_port=29602 \
ChatUniVi/train/train_mem.py \
--deepspeed scripts/zero3.json \
--model_name_or_path ${LLM model path} \
--version v1 \
--model_use PRETUNE \
--dataset_use Pretrain \
--vision_tower openai/clip-vit-large-patch14 \
--tune_mm_mlp_adapter True \
--mm_vision_select_layer -2 \
--mm_use_im_start_end False \
--mm_use_im_patch_token False \
--bf16 True \
--output_dir ${stage1 save path} \
--num_train_epochs 1 \
--per_device_train_batch_size 16 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 24000 \
--save_total_limit 1 \
--learning_rate 2e-3 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--dataloader_num_workers 4 \
--lazy_preprocess True \
--report_to wandb

Stage2: Joint Instruction Tuning

deepspeed \
--include localhost:0,1,2,3,4,5,6,7 \
--master_port=29601 \
ChatUniVi/train/train_mem.py \
--deepspeed scripts/zero2.json \
--model_name_or_path ${LLM model path} \
--version v1 \
--model_use FINETUNE \
--dataset_use FINETUNE \
--vision_tower openai/clip-vit-large-patch14 \
--pretrain_mm_mlp_adapter ${stage1 save path}/mm_projector.bin \
--mm_vision_select_layer -2 \
--mm_use_im_start_end False \
--mm_use_im_patch_token False \
--bf16 True \
--output_dir ${stage2 save path} \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 50000 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--dataloader_num_workers 4 \
--lazy_preprocess True \
--report_to wandb