Enable CB for vlms with multiple images and multiple prompts #583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

quic-hemagnih merged 24 commits into quic:main from quic-mamta:cb_vlms

Nov 4, 2025

Contributor

quic-mamta commented Oct 6, 2025

No description provided.

quic-mamta requested review from ochougul, quic-amitraj, quic-hemagnih and quic-rishinr as code owners

October 6, 2025 06:18

quic-mamta marked this pull request as draft

October 6, 2025 06:19

quic-mamta requested review from asmigosw and mohiso22

October 6, 2025 06:19

quic-mamta self-assigned this

quic-mamta force-pushed the cb_vlms branch from 804d672 to 38e777a Compare

October 7, 2025 11:29

quic-hemagnih mentioned this pull request

Llama4 VLM Continuous Batching Support #510

Closed

quic-rishinr mentioned this pull request

Llama4 VLM Continuous Batching Support #559

Closed

quic-rishinr marked this pull request as ready for review

October 29, 2025 10:10

quic-rishinr force-pushed the cb_vlms branch 2 times, most recently from 8dd6395 to b41bd72 Compare

October 31, 2025 18:28

mamtsing and others added 16 commits

November 3, 2025 10:33


          Enable CB for vlms with multiple images and multiple prompts

1ee373e

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          update text_generation_interface

e9cf657

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated text_generation to run CB for VLMs

f62f71e

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Ruff format

39f5574

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated qwen2_5 modelling for CB support

ecd5905

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated qwen2_5 modelling for CB support

3d9cd49

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Passed image queue in decode CB

165c8fb

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Ruff format

cf2d4fb

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          refactored the code

d97bda9

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Qwen2.5vl CB Update

e663cd6

Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Lint fix and code cleaning

799af59

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          fix for fbs >1

621e3a8

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated cloud_ai_100_exec_kv call in modelling_auto

59dff46

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Removed redundant vision execution and refactoring

fc69e3a

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          nit: update QBlocking to LM Attention in Qwen2.5VL

275d4cd

Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          nit: update readme for qblocking in example file and lint/format checks

e8253ff

Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

Vinayak Baddi and others added 5 commits

November 3, 2025 10:33


          nit: lint/format checks

4d9afe2

Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          CI failure fix

6b079fe

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          qwen2_5_vl inference changes

64dffdd

Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Removed CB regard kw args for functionin non CB models

ca0cc03

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Added caching for vision outputs

b0ee5a4

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

quic-rishinr force-pushed the cb_vlms branch from b41bd72 to b0ee5a4 Compare

November 3, 2025 05:03

quic-dhirajku reviewed

View reviewed changes

QEfficient/generation/embedding_handler.py Show resolved Hide resolved


          Added change for poping prompt while running continuous batching

a0a80c0

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

quic-amitraj requested changes

View reviewed changes

QEfficient/transformers/models/modeling_auto.py Outdated Show resolved Hide resolved

QEfficient/transformers/models/modeling_auto.py Show resolved Hide resolved

QEfficient/transformers/models/modeling_auto.py Show resolved Hide resolved

QEfficient/transformers/models/modeling_auto.py Show resolved Hide resolved

quic-amitraj reviewed

View reviewed changes

QEfficient/generation/vlm_generation.py Show resolved Hide resolved


          Adding multi_frame modeling changes and some fix

Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>

quic-amitraj requested changes

View reviewed changes

QEfficient/generation/vlm_generation.py Outdated Show resolved Hide resolved

QEfficient/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py Show resolved Hide resolved


          Adressed review comments

875c4d2

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

quic-hemagnih approved these changes

View reviewed changes

QEfficient/generation/cloud_infer.py

    
                      del self.execObj

                      self.program.deactivate()

                      if self.is_active:

                          del self.execObj

Contributor

quic-hemagnih Nov 4, 2025

NIT pick, we should always have a cleanup method declared for execObj, else it may lead to silent memory leaks. Just deleting the object may lead to hidden memory leaks.

quic-amitraj approved these changes

View reviewed changes

Contributor

quic-amitraj left a comment

LGTM

asmigosw approved these changes

View reviewed changes

Contributor

asmigosw left a comment

LGTM

quic-hemagnih merged commit 7e8838f into quic:main

5 checks passed

abhishek-singh591 pushed a commit to abhishek-singh591/quic_abhishek that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

f82d563

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>

vjanfaza pushed a commit to vjanfaza/efficient-transformers that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

20f1c71

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>

qraniumcitest mentioned this pull request

Qgenie Prompt testing qraniumcitest/efficient-transformers#46

Closed

abhishek-singh591 pushed a commit to quic-rishinr/efficient-transformers that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>
Signed-off-by: abhishek-singh591 <sabhis@qti.qualcomm.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

quic-dhirajku quic-dhirajku left review comments

quic-amitraj quic-amitraj approved these changes

quic-hemagnih quic-hemagnih approved these changes

quic-rishinr Awaiting requested review from quic-rishinr quic-rishinr is a code owner

ochougul Awaiting requested review from ochougul ochougul is a code owner

mohiso22 Awaiting requested review from mohiso22

+1 more reviewer

asmigosw asmigosw approved these changes

Labels

None yet