Add inference request cancellation APIs #249
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tabrizian
force-pushed
the
imant-cancellation-api
branch
from
a2d565a
to
73a923d
Compare
nnshah1
reviewed
Aug 30, 2023
|
can we add the error code for cancellation into core/include/triton/core/tritonserver.h Line 311 in b20f7b5
|
|
@nnshah1 sorry, I missed that. Added cancellation status. |
nnshah1
reviewed
Aug 31, 2023
Tabrizian
force-pushed
the
imant-cancellation-api
branch
from
83c0077
to
1541211
Compare
Tabrizian
force-pushed
the
imant-cancellation-api
branch
from
1541211
to
2aabb8b
Compare
nnshah1
reviewed
Sep 6, 2023
nnshah1
approved these changes
Sep 6, 2023
Tabrizian
added a commit
that referenced
this pull request
Sep 13, 2023
* Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Tabrizian
added a commit
that referenced
this pull request
Sep 13, 2023
* Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Tabrizian
added a commit
that referenced
this pull request
Sep 13, 2023
* Add inference request cancellation APIs (#249) * Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Fix request re-use when cancelling a request * Review edit * Fix warmup request * Fix null requests * Review edit --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
tanmayv25
added a commit
that referenced
this pull request
Oct 4, 2023
* Add inference request cancellation APIs (#249) * Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Fix request re-use when cancelling a request (#253) * Add inference request cancellation APIs (#249) * Fix state transitions for re-running requests (#251) * Add backend/server APIs * Implement the cancellation APIs * Only store the state in response factory * Add unit testing for request cancellation * Add test * Add cancellation status * Add testing for cancelling a request after release * Handle request re-use * Enable request reuse test * Add staged changes * Add temporary fix for the request state bug --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Fix request re-use when cancelling a request * Review edit * Fix warmup request * Fix null requests * Review edit --------- Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> * Dynamic batch scheduler request cancellation (#257) * Add adapter for IsCancelled() * Move cancel functions outside TRITON_ENABLE_STATS * Add request cancellation to dynamic batch scheduler * Refactor cancelled requests to use rejected requests routine * Remove shared pointer wrapper for rejected and cancelled requests * Ensemble scheduler request cancellation (#263) * Add adapter for IsCancelled() * Move cancel functions outside TRITON_ENABLE_STATS * Add request cancellation to ensemble scheduler * Sequence batch scheduler request cancellation (#260) * Add request cancellation to sequence batch scheduler * Add request cancellation to in-flight sequences * Refactor on how a request is cancelled * Fix issue when request is timeout dummy * Always mark request cancelled before cancelling * Mark immutable static status as const --------- Co-authored-by: Iman Tabrizian <iman.tabrizian@gmail.com> Co-authored-by: Ryan McCormick <rmccormick@nvidia.com> Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
server: triton-inference-server/server#6252
common: triton-inference-server/common#99