Description
Hi,
As known from the detectron2 deployment description, the detectron2 TorchScript scripting model supports dynamic batch_size. I am currently working on modifying the official example "[torchscript_mask_rcnn.cpp]" into batch inference with batch_size>1. However, it does not works.
//create a Tuple[Dict[str, Tensor]] which is the input type of scripted model
c10::IValue get_scripting_inputs(cv::Mat& img, c10::Device device) {
const int height = img.rows;
const int width = img.cols;
const int channels = 3;
auto img_tensor =
torch::from_blob(img.data, { height, width, channels }, torch::kUInt8);
// HWC to CHW
img_tensor =
img_tensor.to(device, torch::kFloat).permute({ 2, 0, 1 }).contiguous();
cout << "img_tensor" << img_tensor.sizes() << endl;
auto img_tensor_l = img_tensor.unsqueeze(0);
cout << "img_tensor_l" << img_tensor_l.sizes() << endl;
auto dic = c10::Dict<std::string, torch::Tensor>();
dic.insert("image", img_tensor);
return std::make_tuple(dic);
}
1, The example mentions "create a Tuple[Dict[str, Tensor]] which is the input type of scripted model", so I have tried to create a
Tuple[Dict[str, Tensor], Dict[str, Tensor]]
by using
return std::make_tuple(dic, dic)
in the return of the above function get_scripting_inputs. However, it reports the errors
"Expected a value of type 'Tuple[Dict[str, Tensor]]' for argument 'inputs' but instead found type 'Tuple[Dict[str, Tensor], Dict[str, Tensor]]'."
2, Alternatively, I have tried to stack the inputs as:
auto inputs = get_scripting_inputs(input_img_resize, device);
std::vector<c10::IValue> inputs_list;
for (int i = 0; i < 5; i++) {
inputs_list.push_back(inputs);
}
c10::Stack stack{ inputs_list };
auto outputs = model.forward({ stack });
However, it gives the error "Expected at most 2 argument(s) for operator 'forward', but received 6 argument(s). Declaration: forward(torch.ScriptableAdapter self, (Dict(str, Tensor)) inputs) -> (Dict(str, Tensor)[])".
Can anyone provide some advices on how to realize the batch inference on the Torchscript scripting? Many Thanks.