VideoMAEForVideoClassification does not support `device_map='auto'` yet.

[MichaelRipa]

Feature request

Support for device_map = 'auto' so that the VideoMAE models can be run with Int8 mixed precision. For reproducibility, here is what I get when I run the command in a collab notebook (w/ GPU) with accelerate and bitsandbytes installed:

from transformers import AutoModelForVideoClassification

model_name = 'MCG-NJU/videomae-base-finetuned-ssv2 #Example checkpoint
model = AutoModelForVideoClassification.from_pretrained(model_name,load_in_8bit=True,device_map='auto')

Which gives the following error message:

Overriding torch_dtype=None with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 4>:4                                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:471 in          │
│ from_pretrained                                                                                  │
│                                                                                                  │
│   468 │   │   │   )                                                                              │
│   469 │   │   elif type(config) in cls._model_mapping.keys():                                    │
│   470 │   │   │   model_class = _get_model_class(config, cls._model_mapping)                     │
│ ❱ 471 │   │   │   return model_class.from_pretrained(                                            │
│   472 │   │   │   │   pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs,   │
│   473 │   │   │   )                                                                              │
│   474 │   │   raise ValueError(                                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:2703 in from_pretrained   │
│                                                                                                  │
│   2700 │   │   │   )                                                                             │
│   2701 │   │   │                                                                                 │
│   2702 │   │   │   if model._no_split_modules is None:                                           │
│ ❱ 2703 │   │   │   │   raise ValueError(f"{model.__class__.__name__} does not support `device_m  │
│   2704 │   │   │   no_split_modules = model._no_split_modules                                    │
│   2705 │   │   │   if device_map not in ["auto", "balanced", "balanced_low_0", "sequential"]:    │
│   2706 │   │   │   │   raise ValueError(                                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: VideoMAEForVideoClassification does not support `device_map='auto'` yet.

Motivation

I saw a similar issue #22018 which got resolved really quickly. Hoping that this won't be a lot of work to incorperate into the VideoMAE models 🙂

Your contribution

Would prefer if someone more familiar with the repo did this instead (it doesn't appear to be much work if the update is like #22207 but I didn't understand what the change did and don't currently have time to study the codebase)

[sgugger]

cc @alaradirik and @amyeroberts

[meiqing-wang]

Hi I would like give an update about this question: the questions was solved by manually setting device_map in accelerate.
/
Hi, thanks for the commitment. I tested with this change, but there is still a bug which made me very confusing:

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`

My codes can run on one GPU, and I met this error when I run it on two GPUs with this change.
I already checked the compatibility of CUDA, pytorch etc...Also did small tests about training on other small dataset with two GPUs by simple pytorch codes. They all worked. I even set the batch_size=1, the error is still there...
If you have any idea about this error, I am really appreciated.

[ArthurZucker]

cc @rafaelpadilla if you can take over the PR would be nice 😉

[rafaelpadilla]

I'm on it. I will take a look in this issue.

[rafaelpadilla]

Hi @shiliu0111 ,

Thank you for opening this issue.
I tried to reproduce your error, but everything seems to be working on my side. I tried loading the model by setting
device_map=accelerator.device.

Could you please provide a code snippet that would allow me to replicate your error? Also kindly share the contents of your accelerate/default_config.yaml, so I can use the same configuration as you have.

[meiqing-wang]

Hi @rafaelpadilla,

Thanks for looking into this problem. I have solved this by manually setting up device_map. The codes now work very well.

Hi @shiliu0111 ,

Thank you for opening this issue. I tried to reproduce your error, but everything seems to be working on my side. I tried loading the model by setting device_map=accelerator.device.

Could you please provide a code snippet that would allow me to replicate your error? Also kindly share the contents of your accelerate/default_config.yaml, so I can use the same configuration as you have.

[rafaelpadilla]

Hi @shiliu0111 ,

Happy to hear the problem was solved 😀
I will close this issue for now. You can reopen it in case you encounter any related problems.

[bhaswata08]

Update on this issue, cant implement it still, I am using kaggle notebook with the latest version of transformers,

from transformers import VideoMAEImageProcessor, AutoModelForVideoClassification
model_ckpt = "MCG-NJU/videomae-base"
image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
model = VideoMAEForVideoClassification.from_pretrained(
    model_ckpt,
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
    device_map = 'auto'
)

ERROR:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[28], line 4
      1 from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
      3 image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
----> 4 model = VideoMAEForVideoClassification.from_pretrained(
      5     model_ckpt,
      6     label2id=label2id,
      7     id2label=id2label,
      8 #     load_in_8bit=True,
      9     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
     10     device_map = 'auto'
     11 )

File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:3423, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
   3420 if hf_quantizer is not None:
   3421     target_dtype = hf_quantizer.adjust_target_dtype(target_dtype)
-> 3423 no_split_modules = model._get_no_split_modules(device_map)
   3424 if device_map not in ["auto", "balanced", "balanced_low_0", "sequential"]:
   3425     raise ValueError(
   3426         "If passing a string for `device_map`, please choose 'auto', 'balanced', 'balanced_low_0' or "
   3427         "'sequential'."
   3428     )

File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:1752, in PreTrainedModel._get_no_split_modules(self, device_map)
   1750 if isinstance(module, PreTrainedModel):
   1751     if module._no_split_modules is None:
-> 1752         raise ValueError(
   1753             f"{module.__class__.__name__} does not support `device_map='{device_map}'`. To implement support, the model "
   1754             "class needs to implement the `_no_split_modules` attribute."
   1755         )
   1756     else:
   1757         _no_split_modules = _no_split_modules | set(module._no_split_modules)

ValueError: VideoMAEForVideoClassification does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.