✨[Feature] Custom input tensors

[Fleyderer]

Is your feature request related to a problem? Please describe.
Problem is that models sometimes need special inputs (for example boxes, where first points coordinates (TL) is less than second (BR)), but now there are only random values from 0 to 1, so model cannot be converted.

Describe the solution you'd like
I would like to have ability to pass own tensors, which will be used while compiling. For example, for dynamic_shapes I could pass min-opt-max shapes and my own tensors at the same time.

Describe alternatives you've considered
No any. I've tried to change some source code, but as I can see, final input is made in C++, not in Python, so I can't change it.

Additional context

[gs-olive]

Hi @Fleyderer - thank you for the request. This is related to #1425 which added support for specification of an explicit tensor domain to sample from, but stopped short of fully-user-specified input tensors.

[Fleyderer]

but stopped short of fully-user-specified input tensors.

Everything has already been invented – while converting from PyTorch to ONNX, we pass manually created dummy inputs, so I think that it would be most elegant solution – we can fully control what we pass to model, control shapes and values.

In my situation real problem is that I pass tensor with bounding boxes in TLBR format, so I can't normally export model to TensorRT.

I'm really looking forward to this opportunity

[gs-olive]

Thank you for the follow-up. On further review, I believe this feature is already in progress for the default Dynamo path in #2271. Users will be able to specify torch_tensor along with the shape specification to provide a sample input tensor - @peri044 - is this an accurate usage of that feature?

[peri044]

I would like to have ability to pass own tensors, which will be used while compiling. For example, for dynamic_shapes I could pass min-opt-max shapes and my own tensors at the same time.

Hello @Fleyderer , in this PR #2271 (which is based on ir=dynamo and not torchscript), we are trying to support the following

input = torch_tensorrt.Input(min_shape=<min_shape>, opt_shape=<opt_shape>, max_shape=<max_shape>, torch_tensor=<user_tensor>)

Would this fit your usecase ? or do you need to pass a specific tensor for min, opt and max separately ? In the latter case, we can probably accept a tuple in the torch_tensor attribute.

[Fleyderer]

Hello @peri044, it depends on what do you pass while converting model to TensorRT. If you pass only one tensor (which is stored in torch_tensor parameter) and don't pass tensors of other shapes, one torch_tensor would be enough. I think, it would be great solution to have another constructor with min/opt/max tensor parameters, where user can pass his tensors and then both shapes and values of these tensors are used in further conversion. Your solution with tuple of tensors is also good, but needs more coding for user (:

Firstly, when I met this problem, I've tried to inherit your Input class and change behavior of example_tensor() method, because I've been thinking that you use this method when pass tensors for conversion.

My usecase is a bit hard for conversion, because I have 3 inputs, where the first one has 3 dynamic axis of 4, second and third has dynamic batch size, but its values should be points of boxes, which are being cropped in for-loop (model is traced, so loop is dynamic in TorchScript). If your solution will allow pass tensors with dynamic shapes but specific values in it, it will be great