Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dlinfer] Fix qwenvl rope error for dlinfer backend #2795

Merged
merged 2 commits into from
Nov 25, 2024
Merged

[dlinfer] Fix qwenvl rope error for dlinfer backend #2795

merged 2 commits into from
Nov 25, 2024

Conversation

JackWeiw
Copy link
Contributor

@JackWeiw JackWeiw commented Nov 22, 2024 •
edited
Loading

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Qwen2_vl is using mrope, the first dimension of mrope_position_ids 3, so we need to expand the first dimension of inv_freq to 3 to match shape constraint for bmm.

@jinminxi104 jinminxi104 changed the title Fix qwenvl rope error Fix qwenvl rope error for dlinfer backend Nov 22, 2024
@jinminxi104 jinminxi104 self-requested a review November 22, 2024 08:52
@jinminxi104 jinminxi104 changed the title Fix qwenvl rope error for dlinfer backend [dlinfer] Fix qwenvl rope error for dlinfer backend Nov 25, 2024
@lvhan028 lvhan028 merged commit f13c0f9 into InternLM:main Nov 25, 2024
5 checks passed
lvhan028 added a commit that referenced this pull request Nov 25, 2024
@lvhan028 @akai-shuuichi
@jinminxi104 @zhulinJulia24 @Reinerzhou @AllentDan @JackWeiw
Resolve conflicts (#2811)
e7b5a7e 
* feature: support qwen2.5 fuction_call (#2737)

* feat: support qwen2.5 tools_call

* fix: npe bug

* fix: 模版不一致

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* feat: Support multi tools calling

* feat: Support multi tools calling

* fix: Add '\n' between each tool

* fix: Add ensure_ascii=False

* bugfix: rfind

* bugfix: tools_call -> tool_calls

* bugfix: add toolName in tool_response

* fix: some '\n' error

* fix: remove toolname

* fix: replace '\n' to self.separator

* feat: add doc with multiple tool calling

* fix：update doc

* feat: add qwen2.5 prompt template test

* feat: add qwen2.5 no tool call prompt test

---------

Co-authored-by: gaozixiang <gaozixiang1@xiaomi.com>

* Update supported models & Ascend doc (#2765)

* update ascend supported model list

* fix markdown

* fix markdown

* fix lint

* Update get_started.md

* Update get_started.md

* [CI] Split vl testcases into turbomind and pytorch backend (#2751)

* updaet

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* [Feature] support minicpm-v_2_6 for pytorch engine. (#2767)

* support minicpmv_2_6.

* update supported_models.

* update supported_models.

* Support qwen2-vl AWQ quantization (#2787)

* Support qwen2-vl AWQ quantization

* Update config.yaml

---------

Co-authored-by: zhulinJulia24 <145004780+zhulinJulia24@users.noreply.github.com>

* [dlinfer] Fix qwenvl rope error for dlinfer backend (#2795)

* Optimize update_step_ctx on Ascend (#2804)

* opt update_ctx for ascend

* fix lint

---------

Co-authored-by: 逝夜长歌 <928926035@qq.com>
Co-authored-by: gaozixiang <gaozixiang1@xiaomi.com>
Co-authored-by: jinminxi104 <jinminxi104@hotmail.com>
Co-authored-by: zhulinJulia24 <145004780+zhulinJulia24@users.noreply.github.com>
Co-authored-by: zhoushenglong <87467364+Reinerzhou@users.noreply.github.com>
Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com>
Co-authored-by: Wei Tao <1136862851@qq.com>
@JackWeiw JackWeiw deleted the fix_qwen branch November 29, 2024 02:37
lvhan028 added a commit that referenced this pull request Dec 13, 2024
@lvhan028 @akai-shuuichi
@jinminxi104 @zhulinJulia24 @Reinerzhou @AllentDan @JackWeiw @grimoire
Refactor VLM modules (#2810)
96e82eb 
* refactor VL modules for internvl and qwen2-vl (#2764)

* qwen2-vl

* internvl

* qwen2

* Refactor VL modules for glm4v, deepseek-vl, llava-hf, cogvlm (#2772)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* Refactor VL modules for qwen-vl, llava and llava_next (#2773)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* Refactor VL modules for qwen2-vl (#2777)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* Fix side-effect to internvl (#2778)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* Refactor VL modules for phi3-vision (#2779)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* phi3-vision

* Refactor VL modules for mllama and yi-vl (#2781)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* phi3-vision

* refactor yi-vl

* refactor mllama

* Refactor VLM module for minicpm and molmo (#2794)

* Refactor VLM modules for xcomposer series (#2796)

* Refactor VLM modules for internvl-llava (#2797)

* Refactor VLM modules v2 (#2806)

* internvl2 v2

* cogvlm

* deepseek-vl

* glm-4v

* llava-hf

* llava-next

* llava

* internvl-llava

* mllama

* phi3-vision

* qwen

* qwen2

* yi-vl

* xcomposer

* minicpm

* molmo

* update

* update

* Remove vl template (#2809)

* Resolve conflicts (#2811)

* feature: support qwen2.5 fuction_call (#2737)

* feat: support qwen2.5 tools_call

* fix: npe bug

* fix: 模版不一致

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* feat: Support multi tools calling

* feat: Support multi tools calling

* fix: Add '\n' between each tool

* fix: Add ensure_ascii=False

* bugfix: rfind

* bugfix: tools_call -> tool_calls

* bugfix: add toolName in tool_response

* fix: some '\n' error

* fix: remove toolname

* fix: replace '\n' to self.separator

* feat: add doc with multiple tool calling

* fix：update doc

* feat: add qwen2.5 prompt template test

* feat: add qwen2.5 no tool call prompt test

---------

Co-authored-by: gaozixiang <gaozixiang1@xiaomi.com>

* Update supported models & Ascend doc (#2765)

* update ascend supported model list

* fix markdown

* fix markdown

* fix lint

* Update get_started.md

* Update get_started.md

* [CI] Split vl testcases into turbomind and pytorch backend (#2751)

* updaet

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* [Feature] support minicpm-v_2_6 for pytorch engine. (#2767)

* support minicpmv_2_6.

* update supported_models.

* update supported_models.

* Support qwen2-vl AWQ quantization (#2787)

* Support qwen2-vl AWQ quantization

* Update config.yaml

---------

Co-authored-by: zhulinJulia24 <145004780+zhulinJulia24@users.noreply.github.com>

* [dlinfer] Fix qwenvl rope error for dlinfer backend (#2795)

* Optimize update_step_ctx on Ascend (#2804)

* opt update_ctx for ascend

* fix lint

---------

Co-authored-by: 逝夜长歌 <928926035@qq.com>
Co-authored-by: gaozixiang <gaozixiang1@xiaomi.com>
Co-authored-by: jinminxi104 <jinminxi104@hotmail.com>
Co-authored-by: zhulinJulia24 <145004780+zhulinJulia24@users.noreply.github.com>
Co-authored-by: zhoushenglong <87467364+Reinerzhou@users.noreply.github.com>
Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com>
Co-authored-by: Wei Tao <1136862851@qq.com>

* PytorchEngine refactor multimodal (#2742)

* WIP

* support mrope

* support long context

* support causal=false

* fix mask

* flash attn bound

* optimize

* Moskau, Moskau, wirf die Gläser an die Wand

* YMCA

* optimize mllama

* update processor

* support cogvlm

* all work and no play make jack a dull boy

* upgrade triton

* support qwen2vl

* support internvl

* phi3-v WIP

* glm4v WIP

* support chatglm and cogvlm

* use image tokens

* support llava

* support internvl-mono

* phi3v, mllama

* add llavanext

* use img token ids

* support multiimage chatglm cogvlm

* fix ut

* minor-fix

* minor-fix (#2813)

* fix

* fix mono

* fix docs

* read norm_type

* super().collect_images->self.collect_images

* add note in supported models

* define the parameters clearly

* better streaming

* fix molmo

* Fix vision model batch inference (#2868)

* remove forward from vl models that are not supported by tm

* support max_batch_size

* fix

* warn glm4v does not support multi images

* unconst

* fix deepseek-vl

* fix internvl

* fix llava

* fix minicpm 2.6

* fix callback

* fix minicpm v2.5

* fix minicpm v2.6

* update llava_next.py

* remove hardcode from xcomposer2.py

* rollback supported_models

* change to staticmethod

* fix vlm quantization

* update doc

* update

---------
Co-authored-by: q yao <streetyao@live.com>
lvhan028 pushed a commit that referenced this pull request Dec 17, 2024
@grimoire
Optimize tp broadcast (#2889)
8afb84c 
* refactor VL modules for internvl and qwen2-vl (#2764)

* qwen2-vl

* internvl

* qwen2

* Refactor VL modules for glm4v, deepseek-vl, llava-hf, cogvlm (#2772)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* Refactor VL modules for qwen-vl, llava and llava_next (#2773)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* Refactor VL modules for qwen2-vl (#2777)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* Fix side-effect to internvl (#2778)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* Refactor VL modules for phi3-vision (#2779)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* phi3-vision

* Refactor VL modules for mllama and yi-vl (#2781)

* qwen2-vl

* internvl

* qwen2

* get image_tokens_per_patch for internvl2

* deepseek-vl

* cogvlm

* glm4v

* update internvl

* internvl_llava

* llava

* glm4v

* upate internvl

* cogvlm

* deepseek

* llava_hf

* rollback llava, internvl-llava

* refactor qwen

* update internvl

* update llava_hf

* update qwen2-vl

* llava_next

* update llava_next

* update llava

* update llava

* update llava

* qwen2

* fix internvl

* phi3-vision

* refactor yi-vl

* refactor mllama

* Refactor VLM module for minicpm and molmo (#2794)

* Refactor VLM modules for xcomposer series (#2796)

* Refactor VLM modules for internvl-llava (#2797)

* Refactor VLM modules v2 (#2806)

* internvl2 v2

* cogvlm

* deepseek-vl

* glm-4v

* llava-hf

* llava-next

* llava

* internvl-llava

* mllama

* phi3-vision

* qwen

* qwen2

* yi-vl

* xcomposer

* minicpm

* molmo

* update

* update

* Remove vl template (#2809)

* Resolve conflicts (#2811)

* feature: support qwen2.5 fuction_call (#2737)

* feat: support qwen2.5 tools_call

* fix: npe bug

* fix: 模版不一致

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* fix: adopting review suggestions

* feat: Support multi tools calling

* feat: Support multi tools calling

* fix: Add '\n' between each tool

* fix: Add ensure_ascii=False

* bugfix: rfind

* bugfix: tools_call -> tool_calls

* bugfix: add toolName in tool_response

* fix: some '\n' error

* fix: remove toolname

* fix: replace '\n' to self.separator

* feat: add doc with multiple tool calling

* fix：update doc

* feat: add qwen2.5 prompt template test

* feat: add qwen2.5 no tool call prompt test

---------

Co-authored-by: gaozixiang <gaozixiang1@xiaomi.com>

* Update supported models & Ascend doc (#2765)

* update ascend supported model list

* fix markdown

* fix markdown

* fix lint

* Update get_started.md

* Update get_started.md

* [CI] Split vl testcases into turbomind and pytorch backend (#2751)

* updaet

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* [Feature] support minicpm-v_2_6 for pytorch engine. (#2767)

* support minicpmv_2_6.

* update supported_models.

* update supported_models.

* Support qwen2-vl AWQ quantization (#2787)

* Support qwen2-vl AWQ quantization

* Update config.yaml


* [dlinfer] Fix qwenvl rope error for dlinfer backend (#2795)

* Optimize update_step_ctx on Ascend (#2804)

* opt update_ctx for ascend

* fix lint


* PytorchEngine refactor multimodal (#2742)

* WIP

* support mrope

* support long context

* support causal=false

* fix mask

* flash attn bound

* optimize

* Moskau, Moskau, wirf die Gläser an die Wand

* YMCA

* optimize mllama

* update processor

* support cogvlm

* all work and no play make jack a dull boy

* upgrade triton

* support qwen2vl

* support internvl

* phi3-v WIP

* glm4v WIP

* support chatglm and cogvlm

* use image tokens

* support llava

* support internvl-mono

* phi3v, mllama

* add llavanext

* use img token ids

* support multiimage chatglm cogvlm

* fix ut

* minor-fix

* minor-fix (#2813)

* fix

* fix mono

* fix docs

* read norm_type

* super().collect_images->self.collect_images

* add note in supported models

* define the parameters clearly

* better streaming

* fix molmo

* Fix vision model batch inference (#2868)

* remove forward from vl models that are not supported by tm

* support max_batch_size

* fix

* warn glm4v does not support multi images

* unconst

* fix deepseek-vl

* fix internvl

* fix llava

* fix minicpm 2.6

* fix callback

* fix minicpm v2.5

* fix minicpm v2.6

* update llava_next.py

* remove hardcode from xcomposer2.py

* rollback supported_models

* change to staticmethod

* optimize tp

* fix vlm quantization

* update doc

* update
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jinminxi104 jinminxi104 jinminxi104 approved these changes

@CyCle1024 CyCle1024 CyCle1024 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@JackWeiw @CyCle1024 @jinminxi104 @lvhan028