-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[wip] NPUW: Handle closures just once for single-shot functions #27132
base: master
Are you sure you want to change the base?
Conversation
// So - closure update is NOT required, OR the function is SINGLE - | ||
// just handle it's closure here and don't do it in runtime | ||
if (!m_closure_update_required || func_desc.forced_to_fcall) { | ||
unpack_closure(i, m_subrequests[real_idx]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it'll be called in CWAI case or in forced case. Why do we change set_tensor
to unpack_closure
for CWAI case?
@@ -718,7 +707,7 @@ void ov::npuw::JustInferRequest::function_prologue(std::size_t idx) { | |||
// 2. Unpack the function closure -- right here, if pipelining if not enabled. | |||
// If it is enabled, the flow is a little bit different - see run_subrequest_for_success() | |||
// for details. | |||
if (!is_pipelined(idx)) { | |||
if (!is_pipelined(idx) && m_closure_update_required && !func_desc.forced_to_fcall) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait... is_pipelined
checks && !desc.forced_to_fcall
- how does it work with && !func_desc.forced_to_fcall
?
@@ -765,7 +754,7 @@ void ov::npuw::JustInferRequest::unpack_closure(std::size_t idx, RqPtr request) | |||
if (closure.get_element_type() != clparam->get_element_type()) { | |||
// Remember where the unpack is required | |||
closure_unpack_required.push_back(cidx); | |||
} else if (m_npuw_model->m_update_required) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you explain this?
Before we had
- CWAI (forced to fcall) - set_tensor
- Ordinary rep block, not forced - copy
- Single block, forced - unpack
Now
- CWAI (forced to fcall) - unpack
- Ordinary rep block, not forced - unpack
- Single block, forced - copy
Right?
Details:
Tickets: