Skip to content

Improve the TFTRT optimization path #1880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 10, 2020
Merged

Improve the TFTRT optimization path #1880

merged 2 commits into from
Aug 10, 2020

Conversation

tanmayv25
Copy link
Contributor

@tanmayv25 tanmayv25 commented Aug 7, 2020

Not running CI as the changes to this file will not show impact right away...
I am not authenticated to push to the tensorflow repository. Filing here to collect reviews.

However, there is still the matter of some ops not getting recognized as TF-TRT Candidates for optimization purposes, but are included when the network is converted outside triton. I am waiting for TFTRT team response to understand the same.

These changes fix the problem. I was missing one of the libraries in container. These changes improve the performance of optimized TRTEngine by 25% as seen on resent_v1_50_savedmodel.

Before

2020-08-05 00:21:41.520011: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:486] There are 16 ops of 9 different types in the graph that are not converted to TensorRT: Softmax, Squeeze, Mean, DataFormatDimMap, Add, NoOp, Placeholder, Pad, DataFormatVecPermute, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:633] Number of TensorRT candidate segments: 8

After (Identical behaviour with outside triton)

2020-08-07 18:25:20.833243: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:486] There are 4 ops of 3 different types in the graph that are not converted to TensorRT: Softmax, NoOp, Placeholder, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2020-08-07 18:25:20.854810: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:647] Number of TensorRT candidate segments: 1

@GuanLuo
Copy link
Contributor

GuanLuo commented Aug 7, 2020

I wonder if we are missing something in linking the libraries, is there a way to figure out how are the TF-TRT Candidates being determined?

@tanmayv25
Copy link
Contributor Author

I receive the following reason for Squeeze op not getting selected as candidate:

2020-08-05 00:21:41.519967: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:456] Not a TF-TRT candidate, (Op type: Squeeze), (Op name: import/resnet_v1_50/SpatialSqueeze), (Reason: Invalid argument: Dimension 2 with size -1 cannot be squeezed because it must be size 1, at import/resnet_v1_50/SpatialSqueeze)

I was quite sure these additional constant folding layers would solve the problem by populating these wildcards.

@deadeyegoodwin deadeyegoodwin merged commit 5cce1cf into master Aug 10, 2020
@deadeyegoodwin deadeyegoodwin deleted the tanmayv-tftrt branch August 10, 2020 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants