[ONNXIFI] Top level task for complete ONNXIFI support

[rdzhabarov]

Introduction

There are two ways to execute neural nets through the Glow compiler:

• Use Glow as a stand alone compiler and load Caffe2/ONNX models, see, ImageClassifier for example
• Make Glow embedded into Pytorch/Caffe2 via ONNXIFI interface

The purpose of this issue is to cover completed work for ONNXIFI support, but more importantly outline future plans.

Current state

• At this point we've made a lot of progress and can execute CV models, see, Resnet50 support.
• More sophisticated models which involves various operators can be executed as well, see, list of related closed issues here.
• Support of concurrent execution was added allowing to throttle incoming Pytorch/Caffe2 concurrency to concurrency level supported by a specific Glow backend.

Future work

• Stability and error handling is one of the most important aspects that needs to be in place
• Execution of quantized int8 and fp16 models through the ONNXIFI interface
• Improved debugging experience, per operator logging/statistics
• More to come :)

[yinghai]

More issue to consider: #2071

[nickgg]

@jackm321 @rdzhabarov feel free to reopen if this is useful.

[jgong5]

A question: is this the (only) way how PyTorch would be integrated with GLOW? If so, would the support of PyTorch training model depend on ONNX training support?

[yinghai]

@jgong5 We are working on integration of PyTorch/Glow. Stay tuned.

It would be helpful to have an end-to-end, standalone ONNXIFI example.