feat: Implement Dynamic shapes + fallback support for export path #2271

peri044 · 2023-08-28T09:46:14Z

Description

Implement Dynamic shapes + fallback support for export path

TLDR:

We need torch inputs at a few places in the codebase and to perform shape analysis. We also need torch_tensorrt.Input objects to hold the dynamic shape information accordingly for graph/subgraph inputs. Input.torch_tensor attribute is created to handle this and utility functions get_torch_inputs can be used. prepare_inputs can only return torchtrt_inputs now. Ideally, I think it would be convenient if Input objects are used through out the code and get_torch_inputs can be used whenever necessary.
I added support for kDouble enums. We support truncation and accept graphs which have torch.float64 as inputs and casts are added internally. But our Input object errors out if the dtype is torch.float64.
Updated aten_tracer to use constraints to set bounds on Input dimensions. These bounds are dictated by TensorRT. The dynamic dimensions cannot be unbounded.

Fixes # (issue)

Type of change

Please delete options that are not relevant and/or add your own.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes
I have added the relevant labels to my PR in so that relevant reviewers are notified

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive

Added a small comment on documentation. Otherwise looks good to me!

gs-olive · 2023-10-02T17:57:28Z

docsrc/user_guide/dynamic_shapes.rst

Add a reference to this document in index.rst to have it reflect on the webpage

done. thanks

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

peri044 and others added 30 commits August 8, 2023 10:45

feat: Move tracing to use aot export apis

f1f202e

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: minor changes

abaf047

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: minor changes

bb1f3cf

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: Rebase with main

3d05b4d

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: rebase

8d99be5

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: minor logging updates

0aad214

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

feat: Add preliminary support for freezing tensors in Dynamo

8899735

fix: Refactor tensor freezing in Dynamo

8af2627

Key op fixes for failing tests

f6969be

fix: Add constant folding utility to freezing

bad1594

chore: Move to new export APIs

db56dd6

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: rebase with dynamo_tensor_freeze branch

bf961f5

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

feat: Add preliminary support for freezing tensors in Dynamo

b13aa82

fix: Refactor tensor freezing in Dynamo

dd95620

Key op fixes for failing tests

6bd3c64

fix: Add constant folding utility to freezing

248073f

feat: Add preliminary support for freezing tensors in Dynamo

3e5f434

fix: Refactor tensor freezing in Dynamo

6bf6945

Key op fixes for failing tests

3b6e1e7

fix: Add constant folding utility to freezing

2107d8e

chore: add BERT test case

fd5a41e

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: remove pdb

f047651

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

feat: Implement dynamic shapes feature

de9795d

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: rebase

ab76c0d

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

chore: minor update

5f2a4f3

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

Merge branch 'export_prototype' into dyn_export

566fbb0

chore: refactor

4949549

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

feat: Add preliminary support for freezing tensors in Dynamo

e4df382

fix: Refactor tensor freezing in Dynamo

d022f4a

Key op fixes for failing tests

9610ba7

chore: rebase with main

708ac64

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Sep 25, 2023

View reviewed changes

chore: update docs

9bcaf49

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot added the documentation Improvements or additions to documentation label Sep 26, 2023

github-actions bot approved these changes Sep 26, 2023

View reviewed changes

chore: update docs

6704cb7

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Sep 26, 2023

View reviewed changes

chore: update docs

560c779

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Sep 26, 2023

View reviewed changes

chore: rebase

03f5f2d

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Sep 30, 2023

View reviewed changes

chore: fix tests

0349810

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Oct 1, 2023

View reviewed changes

chore: updates

912fcab

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Oct 2, 2023

View reviewed changes

chore: revert harness tracer changes

31c09b2

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Oct 2, 2023

View reviewed changes

gs-olive reviewed Oct 2, 2023

View reviewed changes

chore: address review comments

9f0a589

Signed-off-by: Dheeraj Peri <peri.dheeraj@gmail.com>

github-actions bot approved these changes Oct 2, 2023

View reviewed changes

peri044 merged commit 6571252 into main Oct 2, 2023

feat: Implement Dynamic shapes + fallback support for export path #2271

feat: Implement Dynamic shapes + fallback support for export path #2271

Uh oh!

Conversation

peri044 commented Aug 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist:

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

gs-olive left a comment

Choose a reason for hiding this comment

Uh oh!

gs-olive Oct 2, 2023

Choose a reason for hiding this comment

Uh oh!

peri044 Oct 2, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

peri044 commented Aug 28, 2023 •

edited

Loading