[Topi & Relay] Add quantization support for the vision transform model in GPU #7814
Conversation
|
Thanks for the reviewer. We will keep updating more results for other ViT models and contributing more quantization calibration algorithms. |
There was a problem hiding this comment.
Thanks! @huochaitiantang
This PR looks great! I have only few comments.
| if not os.path.exists(logfile): | ||
| os.system("wget https://github.com/TheGreatCold/tvm-vit/raw/master/{}".format(logfile)) | ||
| if not os.path.exists(onnx_path): | ||
| os.system("wget https://github.com/TheGreatCold/tvm-vit/raw/master/{}".format(onnx_path)) |
There was a problem hiding this comment.
As a unit test, I'm thinking that this may not be so good to involve a resource outside. (Network problem or changes to the tvm-vit repo may break the UT. At least use git commit instad of branch like: https://github.com/TheGreatCold/tvm-vit/raw/d2aa1e60eef42e2fdedbd1e13aa85ac5faf0a7fc/vit_B32_224.onnx will be better)
I'm not sure if there's any better solution for this. @tqchen Do you have any suggestion?
There was a problem hiding this comment.
Thanks for your review. We updated the download codes based on your suggestion. Besides, the wget method is not compatible on different platforms, so we use the urllib library instead.
There was a problem hiding this comment.
Thanks! @huochaitiantang @XHPlus . Have fun with TVM. Looking forward to see more contributions from you!
…l in GPU (apache#7814) * Add cuda batch matmul int8 support for quantized vit model * Fix for combine parallel pass with dense and batch_matmul * Reformat based on lint * Add plevel & update the file download method
…l in GPU (apache#7814) * Add cuda batch matmul int8 support for quantized vit model * Fix for combine parallel pass with dense and batch_matmul * Reformat based on lint * Add plevel & update the file download method
…l in GPU (apache#7814) * Add cuda batch matmul int8 support for quantized vit model * Fix for combine parallel pass with dense and batch_matmul * Reformat based on lint * Add plevel & update the file download method
…l in GPU (apache#7814) * Add cuda batch matmul int8 support for quantized vit model * Fix for combine parallel pass with dense and batch_matmul * Reformat based on lint * Add plevel & update the file download method
We submit this PR to add quantization support for the vision transform (vit) model in GPU. The main change is as follows:
1, In vit model, time-consuming operators are batch_matmul, so we first add the compute and schedule of
batch_matmul_int8.cudain tvm.topi.cuda2, To support the quantization of batch_matmul, we then add
batch_matmul_rewriteandBatchMatmulRealizein tvm.relay.quantize3, The kl -divergence calibrate could not preserve the accuracy of vit model well, so we add the
_percentile_scalefunctionFor the vit-B32-224 model, the performance is as follows:
Top-1 accuracy in Imagenet validation
The latency in GTX1660 GPU
Thanks for your review! @jcf94 @tqchen