[Feature] [Urgent] Internvl3 requires which versions for trl, transformers, flash_attn and torch

[mustansarfiaz]

Motivation

Hi,

I appreciate your great work on interval 3. I am trying to configure the train setup for interval 3. I am facing dependencies among different library versions, such as torch, trl, and transformers. and flash_attn.
Could you provide which versions you used?

Also, can you share your Dockerfile or link from your Docker hub so that I can use your image?
I am trying to fine-tune internvl_chat/shell/internvl3.0/mpo/internvl3_8b_mpo.sh on my custom dataset.

Regards:
Mustansar

Related resources

No response

Additional context

No response

[lll2343]

Hi，

torch 2.5.1
trl 0.10.1
transformers 4.45.1
flash-attn 2.7.0.post2

[mustansarfiaz]

Hi

Thanks for the response. I installed the mentioned packages versions. Now, I am facing missing --loss_type arguments error.
Here is what I am facing error

[rank0]: Traceback (most recent call last):
[rank0]:   File "./InternVL/internvl_chat/internvl/train/internvl_chat_pretrain.py", line 1116, in <module>
[rank0]:     main()
[rank0]:   File "./InternVL/internvl_chat/internvl/train/internvl_chat_pretrain.py", line 859, in main
[rank0]:     model_args, data_args, training_args = parser.parse_args_into_dataclasses()
[rank0]:   File "./internvl3_env/lib/python3.10/site-packages/transformers/hf_argparser.py", line 348, in parse_args_into_dataclasses
[rank0]:     raise ValueError(f"Some specified arguments are not used by the HfArgumentParser: {remaining_args}")
[rank0]: ValueError: Some specified arguments are not used by the HfArgumentParser: ['--loss_type', 'sigmoid,bco_pair', '--sigmoid_loss_weight', '0.8', '--bco_pair_loss_weight', '0.2', '--rpo_alpha', '1']
[rank0]:[W516 17:45:57.478027427 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())

[mustansarfiaz]

Hi
Can you please share your environment.yaml file?
I am still facing issues related to transformers.

I notice that internvl_chat_pretrain.py works fine for internvl2_5 but fails for internvl3.
I want to train my custom dataset.
Please guide.

Regards
Mustansar

[mustansarfiaz]

Hi

I am trying to finetune the internvl3 using internvl/train/internvl_chat_finetune.py
Here is my loss computation with transformers==4.45.1.
we notice that here loss=0 and grad_norm=1 and model save gives errors.

 96%|█████████▌| 48/50 [00:43<00:01,  1.83it/s]dynamic ViT batch size: 2, images per sample: 2.0, dynamic token length: 12288
[2025-05-17 18:34:29,849] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 15.48 | optimizer_step: 25.82
[2025-05-17 18:34:29,849] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd_microstep: 114.67 | bwd_microstep: 342.42 | bwd_inner_microstep: 325.82 | bwd_allreduce_microstep: 16.54 | step_microstep: 65.26
[2025-05-17 18:34:29,849] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd: 114.66 | bwd: 342.43 | bwd_inner: 325.82 | bwd_allreduce: 16.56 | step: 65.26
{'loss': 0.0, 'grad_norm': 1.0, 'learning_rate': 2.1410767613965208e-10, 'epoch': 0.98}
 98%|█████████▊| 49/50 [00:43<00:00,  1.84it/s]petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
dynamic ViT batch size: 2, images per sample: 2.0, dynamic token length: 12288
[2025-05-17 18:34:31,910] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 16.49 | optimizer_step: 25.88
[2025-05-17 18:34:31,911] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd_microstep: 128.83 | bwd_microstep: 472.35 | bwd_inner_microstep: 430.06 | bwd_allreduce_microstep: 42.20 | step_microstep: 82.90
[2025-05-17 18:34:31,912] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd: 128.81 | bwd: 472.38 | bwd_inner: 430.06 | bwd_allreduce: 42.23 | step: 82.91
{'loss': 0.0, 'grad_norm': 1.0, 'learning_rate': 0.0, 'epoch': 1.0}
100%|██████████| 50/50 [00:46<00:00,  1.00s/it][INFO|trainer.py:2505] 2025-05-17 18:34:31,930 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 46.0759, 'train_samples_per_second': 1.085, 'train_steps_per_second': 1.085, 'train_loss': 0.104375, 'epoch': 1.0}
100%|██████████| 50/50 [00:46<00:00,  1.09it/s]
[INFO|trainer.py:3705] 2025-05-17 18:34:33,174 >> Saving model checkpoint to ./Checkpoints/Internvl3_Exps/internvl_chat_v3_mpo/InternVL3-1B-Instruct_t0_4_45_1
[rank0]: Traceback (most recent call last):
[rank0]:   File "/l/users/mustansar.fiaz/codes/InternVL/internvl_chat/internvl/train/internvl_chat_finetune.py", line 1075, in <module>
[rank0]:     main()
[rank0]:   File "/l/users/mustansar.fiaz/codes/InternVL/internvl_chat/internvl/train/internvl_chat_finetune.py", line 1061, in main
[rank0]:     trainer.save_model()  # Saves the tokenizer too for easy upload
[rank0]:   File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/transformers/trainer.py", line 3610, in save_model
[rank0]:     self._save(output_dir, state_dict=state_dict)
[rank0]:   File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/transformers/trainer.py", line 3727, in _save
[rank0]:     self.model.save_pretrained(
[rank0]:   File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2616, in save_pretrained
[rank0]:     misplaced_generation_parameters = model_to_save.config._get_non_default_generation_parameters()
[rank0]:   File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/transformers/configuration_utils.py", line 1050, in _get_non_default_generation_parameters
[rank0]:     if default_config is not None:
[rank0]: UnboundLocalError: local variable 'default_config' referenced before assignment
[rank0]:[W517 18:34:33.270643220 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
E0517 18:34:36.092000 2547432 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 2547436) of binary: /l/users/mustansar.fiaz/my_environments/internvl3_env/bin/python3.10
Traceback (most recent call last):
  File "/l/users/mustansar.fiaz/my_environments/internvl3_env/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/l/users/mustansar.fiaz/my_environments/internvl3_env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-05-17_18:34:36
  host      : gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 2547436)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
(/l/users/mustansar.fiaz/my_environments/internvl3_env)  1  gpu-01:/l/users/mustansar.fiaz/codes/InternVL/internvl_chat $

Regards
Mustansar

[mustansarfiaz]

Hi

Whereas, if I run for internvl2_5 for internvl/train/internvl_chat_finetune.py
training looks fine and saves the model. I tried with transformers= 4.44.0 Please see the log. Please guide me to fix it for internvl3 for internvl_chat_finetune.py

{'loss': 0.9974, 'grad_norm': 32.78750228881836, 'learning_rate': 1.7110277252379238e-07, 'epoch': 0.96}
 96%|█████████▌| 24/25 [00:33<00:00,  2.65it/s]dynamic ViT batch size: 2, images per sample: 2.0, dynamic token length: 584
[2025-05-17 19:03:36,362] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd_microstep: 52.69 | bwd_microstep: 91.63 | bwd_inner_microstep: 91.60 | bwd_allreduce_microstep: 0.01 | step_microstep: 0.03
Warning: The cache directory for DeepSpeed Triton autotune, /home/mustansar.fiaz/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /home/mustansar.fiaz/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /home/mustansar.fiaz/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
Warning: The cache directory for DeepSpeed Triton autotune, /home/mustansar.fiaz/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
petrel_client is not installed. Using PIL to load images.
dynamic ViT batch size: 2, images per sample: 2.0, dynamic token length: 583
[2025-05-17 19:03:38,318] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 27.95 | optimizer_step: 17.99
[2025-05-17 19:03:38,339] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd_microstep: 183.46 | bwd_microstep: 195.69 | bwd_inner_microstep: 184.71 | bwd_allreduce_microstep: 10.91 | step_microstep: 83.73
[2025-05-17 19:03:38,340] [INFO] [logging.py:107:log_dist] [Rank 0] time (ms) | fwd: 236.12 | bwd: 287.34 | bwd_inner: 276.31 | bwd_allreduce: 10.93 | step: 83.78
{'loss': 0.4305, 'grad_norm': 28.48807716369629, 'learning_rate': 0.0, 'epoch': 1.0}
100%|██████████| 25/25 [00:35<00:00,  1.10it/s][INFO|trainer.py:3513] 2025-05-17 19:03:39,145 >> Saving model checkpoint to ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25
[INFO|configuration_utils.py:472] 2025-05-17 19:03:39,151 >> Configuration saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/config.json
[INFO|configuration_utils.py:807] 2025-05-17 19:03:39,153 >> Configuration saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/generation_config.json
[INFO|modeling_utils.py:2777] 2025-05-17 19:03:41,809 >> Model weights saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/model.safetensors
[INFO|tokenization_utils_base.py:2684] 2025-05-17 19:03:41,814 >> tokenizer config file saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/tokenizer_config.json
[INFO|tokenization_utils_base.py:2693] 2025-05-17 19:03:41,816 >> Special tokens file saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/special_tokens_map.json
[INFO|tokenization_utils_base.py:2744] 2025-05-17 19:03:41,818 >> added tokens file saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/added_tokens.json
[2025-05-17 19:03:42,043] [INFO] [logging.py:107:log_dist] [Rank 0] [Torch] Checkpoint global_step25 is about to be saved!
[2025-05-17 19:03:42,056] [INFO] [logging.py:107:log_dist] [Rank 0] Saving model checkpoint: ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/global_step25/mp_rank_00_model_states.pt
[2025-05-17 19:03:42,056] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/global_step25/mp_rank_00_model_states.pt...
[2025-05-17 19:03:44,382] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/global_step25/mp_rank_00_model_states.pt.
[2025-05-17 19:03:44,383] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/global_step25/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2025-05-17 19:04:01,665] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/global_step25/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2025-05-17 19:04:01,680] [INFO] [engine.py:3701:_save_zero_checkpoint] zero checkpoint saved ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/checkpoint-25/global_step25/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
[2025-05-17 19:04:01,686] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step25 is ready now!
[INFO|trainer.py:2404] 2025-05-17 19:04:01,815 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 59.4128, 'train_samples_per_second': 0.842, 'train_steps_per_second': 0.421, 'train_loss': 1.4982647132873534, 'epoch': 1.0}
100%|██████████| 25/25 [00:59<00:00,  2.38s/it]
[INFO|trainer.py:3513] 2025-05-17 19:04:02,222 >> Saving model checkpoint to ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02
[INFO|configuration_utils.py:472] 2025-05-17 19:04:02,226 >> Configuration saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/config.json
[INFO|configuration_utils.py:807] 2025-05-17 19:04:02,229 >> Configuration saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/generation_config.json
[INFO|modeling_utils.py:2777] 2025-05-17 19:04:04,180 >> Model weights saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/model.safetensors
[INFO|tokenization_utils_base.py:2684] 2025-05-17 19:04:04,193 >> tokenizer config file saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/tokenizer_config.json
[INFO|tokenization_utils_base.py:2693] 2025-05-17 19:04:04,195 >> Special tokens file saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/special_tokens_map.json
[INFO|tokenization_utils_base.py:2744] 2025-05-17 19:04:04,196 >> added tokens file saved in ./Checkpoints/Internvl2_5_Exps/internvl2_5_1b_dynamic_res_2nd_finetune_full_t02/added_tokens.json
***** train metrics *****
  epoch                    =         1.0
  total_flos               = 269936575GF
  train_loss               =      1.4983
  train_runtime            =  0:00:59.41
  train_samples            =          50
  train_samples_per_second =       0.842
  train_steps_per_second   =       0.421
[rank0]:[W517 19:04:04.682102446 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())

[mustansarfiaz]

Hi

@lll2343 @lvhan028 @shepnerd @whai362
when do you plan to release internvl3.0 full finetune and lora finetune scripts?

Regards
Mustansar