Enabled and optimized GLM-4v-9b on Gaudi #691
Conversation
gyou2021
requested review from
kzawora-intel,
madamczykhabana,
michalkuligowski,
mgawarkiewicz,
vivekgoe and
afierka-intel
as code owners
| @@ -1,10 +1,10 @@ | |||
| # Adapted from | |||
There was a problem hiding this comment.
Please revert file mode to 100644, same for other files.
There was a problem hiding this comment.
Other files are in different file modes. It seems that they cannot be reverted to 100644.
vllm/worker/hpu_model_runner.py
Outdated
| @@ -963,7 +961,18 @@ def _prepare_prompt( | |||
| pad=0, | |||
| dtype=torch.long, | |||
| device='cpu') | |||
|
|
|||
| if len(multi_modal_kwargs_list | |||
PatrykWo
added
New Model
|
@gyou2021 Recently vllm has some update about multi-modal support, especially they introduced InputProcessor for each model, like [VLM] Implement merged multimodal processor for Mllama. Will it affect your implementation and will you do a update based on this? |
Signed-off-by: gyou2021 <ganmei.you@intel.com>
No. It won't affect the implementation. So no update based on this is needed. |
Removed graph recompile caused by the image and different batch size;
Example:
python examples/offline_inference_vision_language.py -m glm4v