Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic Batch Support for TRT #6955

Merged
merged 34 commits into from
Dec 1, 2020
Merged

Dynamic Batch Support for TRT #6955

Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a827c10
add_annotate_fn
Nov 23, 2020
7fe57f9
Reshape_ann_fn
Nov 23, 2020
34b5499
Prune Subgraph
Nov 23, 2020
e326d90
Dynamic Shape
Nov 23, 2020
a94c764
Make PT Mask RCNN Work
Nov 24, 2020
08352ab
Cleanup
Nov 24, 2020
8d11a49
Remove comments
Nov 24, 2020
4298de2
Remove COmments
Nov 24, 2020
4d75322
GetBatchSizeFix
Nov 24, 2020
671fca2
Fix Remove Droupout
Nov 24, 2020
a021c4e
Fix Remove Droupout
Nov 24, 2020
408c35f
TRT Runtime
Nov 24, 2020
c70279c
Add MaskrCNN R50
Nov 24, 2020
e38edb0
New Testing code
Nov 24, 2020
1549502
Fix black
Nov 24, 2020
5528c5f
Test Maskrcnn r50 done
Nov 24, 2020
d9366be
Test MR50
Nov 24, 2020
13be588
Space typo
Nov 24, 2020
87c92f8
Change Log to Dlog
Nov 24, 2020
230c125
Move test to tensorrt.py
Nov 25, 2020
d17c4d0
Remove imports
Nov 25, 2020
5bdc028
Remove function
Nov 25, 2020
cab81b7
Add it to trt
Nov 25, 2020
555505d
import error
Nov 25, 2020
ee41e07
Imports
Nov 30, 2020
51b5c8a
Add torch to CI
Nov 30, 2020
6fea658
trt_test
Nov 30, 2020
ea4af58
Check test
Nov 30, 2020
49a4fa7
Revert Pytorch install
Nov 30, 2020
28bc758
Fix
Nov 30, 2020
cf3016b
test dynamic batch
Nov 30, 2020
56c75bf
TRT
Nov 30, 2020
dc6aaef
Resolve PR comments
Nov 30, 2020
121c46b
Zero batch size add
Nov 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
GetBatchSizeFix
  • Loading branch information
Ubuntu committed Nov 24, 2020
commit 4d753223d23c44f0562874d0c6cc66b1353b5f9d
7 changes: 0 additions & 7 deletions src/runtime/contrib/tensorrt/tensorrt_runtime.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,6 @@ class TensorRTRuntime : public JSONRuntimeBase {
void Run() override {
BuildEngine();
batch_size_ = data_entry_[input_var_eid_[0]]->shape[0];

// batch_size_ = GetBatchSize();
auto& engine_and_context = trt_engine_cache_.at(std::make_pair(symbol_name_, batch_size_));
auto engine = engine_and_context.engine;
auto context = engine_and_context.context;
Expand Down Expand Up @@ -178,7 +176,6 @@ class TensorRTRuntime : public JSONRuntimeBase {
* do nothing.
*/
void BuildEngine() {
// batch_size_ = GetBatchSize();
batch_size_ = data_entry_[input_var_eid_[0]]->shape[0];
if (trt_engine_cache_.count(std::make_pair(symbol_name_, batch_size_))) return;
LOG(INFO) << "Building new TensorRT engine for subgraph " << symbol_name_ << " with batch size "
Expand Down Expand Up @@ -262,8 +259,6 @@ class TensorRTRuntime : public JSONRuntimeBase {
* directory so it can be loaded later.
*/
void CacheEngineToDisk() {
// batch_size_ = GetBatchSize();

batch_size_ = data_entry_[input_var_eid_[0]]->shape[0];
std::string cache_dir = dmlc::GetEnv("TVM_TENSORRT_CACHE_DIR", std::string(""));
if (cache_dir.empty()) return;
Expand Down Expand Up @@ -303,8 +298,6 @@ class TensorRTRuntime : public JSONRuntimeBase {
auto nid = input_nodes_[i];
if (nodes_[nid].GetOpType() == "input") {
// Get batch size from first input.

LOG(INFO) << "Current Batch Size " << nodes_[nid].GetOpShape()[0][0];
return nodes_[nid].GetOpShape()[0][0];
}
}
Expand Down