Autocast and Weak Typing

[narendasan]

TL;DR

Weak typing behavior in TensorRT is deprecated. However it is a good way to maximize performance, Therefore we want to create similar PyTorch native system to use with Torch-TensorRT that recovers some of this behavior.

Goal(s)

An automatic pass that users can opt into which will apply autocasting w.r.t. TRT's perfered ruleset.

Usecases

Proposed APIs / UX

1. Support already (PyTorch) autocasted` networks with strong typing
2. Add a lowering pass that applies rules similar to ONNX Autocast for users who want weak typing behavior.

Example Workflow

Limitations

Internal Implementation

Design

Extensions Required to Core API implementations

Data Structures

Details specific for TorchScript Support

Details specific for FX support

Implementation Phases

Prototype -

MVP (<TARGET RELEASE VERSION>)

Extension Phase 1 (<TARGET RELEASE VERSION>)

Extension Phase 2 (<TARGET RELEASE VERSION>)