Fine tuning after stage 1 ckpoint using Llama-2

[sandzp]

I want to fine tune MiniGPT-4 on a custom dataset and LlaMa-2 7b. I was able to successfully fine-tune the model using Vicuna on the same dataset.

I tried using the fine-tune script, with the pretrained model after stage 1 7b and Llama-2-chat-7b from Huggingface:

2023-12-27 15:02:29,449 [INFO] Loading LLAMA
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:37<00:00, 18.52s/it]
2023-12-27 15:04:47,465 [INFO] Loading LLAMA Done
2023-12-27 15:04:47,465 [INFO] Loading VIT
2023-12-27 15:05:08,701 [INFO] freeze vision encoder
2023-12-27 15:05:08,701 [INFO] Loading VIT Done
Do not use Q-Former here.
Load 4 training prompts
Prompt Example 
[INST] <Img><ImageHere></Img> Describe this image in detail. [/INST] 
Load MiniGPT-4 Checkpoint: ../pretrained_minigpt4_7b_stage1.pth
Traceback (most recent call last):
  File "/workspace/MiniGPT-4/train.py", line 104, in <module>
    main()
  File "/workspace/MiniGPT-4/train.py", line 90, in main
    model = task.build_model(cfg)
  File "/workspace/MiniGPT-4/minigpt4/tasks/base_task.py", line 35, in build_model
    return model_cls.from_config(model_config)
  File "/workspace/MiniGPT-4/minigpt4/models/minigpt4.py", line 193, in from_config
    msg = model.load_state_dict(ckpt['model'], strict=False)
  File "/root/miniconda3/envs/minigptv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for MiniGPT4:
	size mismatch for llama_proj.weight: copying a param with shape torch.Size([4096, 768]) from checkpoint, the shape in current model is torch.Size([4096, 5632]).
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 15659) of binary: /root/miniconda3/envs/minigptv/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/envs/minigptv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/minigptv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/minigptv/lib/python3.9/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/root/miniconda3/envs/minigptv/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/root/miniconda3/envs/minigptv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/minigptv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

It seems there is a dimensionality mismatch between the stage 1 pretrained model and that of LlaMa-2, which is making me think that the pretrained weights were those for Vicuna, but not LlaMa?

If this is the case, is there anybody that could provide me with the stage-1 pretrained weights for Llama-2-chat-7b?

Thanks

[phatngodc]

Have you found the answer? I faced the same issue