Skip to content

✨[Feature] Add support for truncate_long_and_double in Dynamo compile #1964

Closed
@gs-olive

Description

@gs-olive

Context

Certain models, including T5 (see #1864, #1740) require INT64 inputs to their embedding layers. 64-bit input types are not compatible with TRT Engines, however, thus a truncation mechanism is necessary to resolve this issue.

Feature Proposal

Add support for truncate_long_and_double in Dynamo compile, tracking data types and automatically casting to TRT-allowed types (and back) as necessary.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions