From 3a7821a93e62ae52f185b937a05ac42ec7ff5ff6 Mon Sep 17 00:00:00 2001 From: Ernestas Date: Wed, 9 Dec 2020 21:41:34 +0200 Subject: [PATCH] build & run instructions --- README.md | 176 ++++++++--------------------------------------- data/config.py | 2 +- export_onnx.sh | 1 + lab.py | 5 ++ requirements.txt | 6 ++ 5 files changed, 43 insertions(+), 147 deletions(-) create mode 100755 export_onnx.sh create mode 100644 requirements.txt diff --git a/README.md b/README.md index 7e2fe7988..3a0423998 100644 --- a/README.md +++ b/README.md @@ -1,146 +1,30 @@ -# **Y**ou **O**nly **L**ook **A**t **C**oefficien**T**s -``` - ██╗ ██╗ ██████╗ ██╗ █████╗ ██████╗████████╗ - ╚██╗ ██╔╝██╔═══██╗██║ ██╔══██╗██╔════╝╚══██╔══╝ - ╚████╔╝ ██║ ██║██║ ███████║██║ ██║ - ╚██╔╝ ██║ ██║██║ ██╔══██║██║ ██║ - ██║ ╚██████╔╝███████╗██║ ██║╚██████╗ ██║ - ╚═╝ ╚═════╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝ ╚═╝ -``` - -A simple, fully convolutional model for real-time instance segmentation. This is the code for [our paper](https://arxiv.org/abs/1904.02689), and for the forseeable future is still in development. - -Here's a look at our current results for our base model (33 fps on a Titan Xp and 29.8 mAP on COCO's `test-dev`): - -![Example 0](data/yolact_example_0.png) - -![Example 1](data/yolact_example_1.png) - -![Example 2](data/yolact_example_2.png) - -# Installation - - Set up a Python3 environment. - - Install [Pytorch](http://pytorch.org/) 1.0.1 (or higher) and TorchVision. - - Install some other packages: - ```Shell - # Cython needs to be installed before pycocotools - pip install cython - pip install opencv-python pillow pycocotools matplotlib - ``` - - Clone this repository and enter it: - ```Shell - git clone https://github.com/dbolya/yolact.git - cd yolact - ``` - - If you'd like to train YOLACT, download the COCO dataset and the 2014/2017 annotations. Note that this script will take a while and dump 21gb of files into `./data/coco`. - ```Shell - sh data/scripts/COCO.sh - ``` - - If you'd like to evaluate YOLACT on `test-dev`, download `test-dev` with this script. - ```Shell - sh data/scripts/COCO_test.sh - ``` - - -# Evaluation -As of April 5th, 2019 here are our latest models along with their FPS on a Titan Xp and mAP on `test-dev`: - -| Image Size | Backbone | FPS | mAP | Weights | | -|:----------:|:-------------:|:----:|:----:|----------------------------------------------------------------------------------------------------------------------|--------| -| 550 | Resnet50-FPN | 42.5 | 28.2 | [yolact_resnet50_54_800000.pth](https://drive.google.com/file/d/1yp7ZbbDwvMiFJEq4ptVKTYTI2VeRDXl0/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/EUVpxoSXaqNIlssoLKOEoCcB1m0RpzGq_Khp5n1VX3zcUw) | -| 550 | Darknet53-FPN | 40.0 | 28.7 | [yolact_darknet53_54_800000.pth](https://drive.google.com/file/d/1dukLrTzZQEuhzitGkHaGjphlmRJOjVnP/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/ERrao26c8llJn25dIyZPhwMBxUp2GdZTKIMUQA3t0djHLw) -| 550 | Resnet101-FPN | 33.0 | 29.8 | [yolact_base_54_800000.pth](https://drive.google.com/file/d/1UYy3dMapbH1BnmtZU4WH1zbYgOzzHHf_/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/EYRWxBEoKU9DiblrWx2M89MBGFkVVB_drlRd_v5sdT3Hgg) -| 700 | Resnet101-FPN | 23.6 | 31.2 | [yolact_im700_54_800000.pth](https://drive.google.com/file/d/1lE4Lz5p25teiXV-6HdTiOJSnS7u7GBzg/view?usp=sharing) | [Mirror](https://ucdavis365-my.sharepoint.com/:u:/g/personal/yongjaelee_ucdavis_edu/Eagg5RSc5hFEhp7sPtvLNyoBjhlf2feog7t8OQzHKKphjw) - -To evalute the model, put the corresponding weights file in the `./weights` directory and run one of the following commands. -## Quantitative Results on COCO -```Shell -# Quantitatively evaluate a trained model on the entire validation set. Make sure you have COCO downloaded as above. -# This should get 29.92 validation mask mAP last time I checked. -python eval.py --trained_model=weights/yolact_base_54_800000.pth - -# Output a COCOEval json to submit to the website or to use the run_coco_eval.py script. -# This command will create './results/bbox_detections.json' and './results/mask_detections.json' for detection and instance segmentation respectively. -python eval.py --trained_model=weights/yolact_base_54_800000.pth --output_coco_json - -# You can run COCOEval on the files created in the previous command. The performance should match my implementation in eval.py. -python run_coco_eval.py - -# To output a coco json file for test-dev, make sure you have test-dev downloaded from above and go -python eval.py --trained_model=weights/yolact_base_54_800000.pth --output_coco_json --dataset=coco2017_testdev_dataset -``` -## Qualitative Results on COCO -```Shell -# Display qualitative results on COCO. From here on I'll use a confidence threshold of 0.3. -python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.3 --top_k=100 --display -``` -## Benchmarking on COCO -```Shell -# Run just the raw model on the first 1k images of the validation set -python eval.py --trained_model=weights/yolact_base_54_800000.pth --benchmark --max_images=1000 -``` -## Images -```Shell -# Display qualitative results on the specified image. -python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.3 --top_k=100 --image=my_image.png - -# Process an image and save it to another file. -python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.3 --top_k=100 --image=input_image.png:output_image.png - -# Process a whole folder of images. -python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.3 --top_k=100 --images=path/to/input/folder:path/to/output/folder -``` -## Video -```Shell -# Display a video in real-time -# I have to work out the kinks for this one. Drawing the frame takes more time than executing the network resulting in sub-30 fps :/ -python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.3 --top_k=100 --video=my_video.mp4 - -# Process a video and save it to another file. -python eval.py --trained_model=weights/yolact_base_54_800000.pth --score_threshold=0.3 --top_k=100 --video=input_video.mp4:output_video.mp4 -``` -As you can tell, `eval.py` can do a ton of stuff. Run the `--help` command to see everything it can do. -```Shell -python eval.py --help -``` - - -# Training - - To train, grab an imagenet-pretrained model and put it in `./weights`. - * For Resnet101, download `resnet101_reducedfc.pth` from [here](https://drive.google.com/file/d/1tvqFPd4bJtakOlmn-uIA492g2qurRChj/view?usp=sharing). - * For Resnet50, download `resnet50-19c8e357.pth` from [here](https://drive.google.com/file/d/1Jy3yCdbatgXa5YYIdTCRrSV0S9V5g1rn/view?usp=sharing). - * For Darknet53, download `darknet53.pth` from [here](https://drive.google.com/file/d/17Y431j4sagFpSReuPNoFcj9h7azDTZFf/view?usp=sharing). - - Run one of the training commands below. - * Note that you can press ctrl+c while training and it will save an `*_interrupt.pth` file at the current iteration. - * All weights are saved in the `./weights` directory by default with the file name `__.pth`. -```Shell -# Trains using the base config with a batch size of 8 (the default). -python train.py --config=yolact_base_config - -# Trains yolact_base_config with a batch_size of 5. For the 550px models, 1 batch takes up around 1.5 gigs of VRAM, so specify accordingly. -python train.py --config=yolact_base_config --batch_size=5 - -# Resume training yolact_base with a specific weight file and start from the iteration specified in the weight file's name. -python train.py --config=yolact_base_config --resume=weights/yolact_base_10_32100.pth --start_iter=-1 - -# Use the help option to see a description of all available command line arguments -python train.py --help -``` - - - -# Citation -If you use YOLACT or this code base in your work, please cite -``` -@article{bolya-arxiv2019, - author = {Daniel Bolya and Chong Zhou and Fanyi Xiao and Yong Jae Lee}, - title = {YOLACT: {Real-time} Instance Segmentation}, - journal = {arXiv}, - year = {2019}, -} -``` - - - -# Contact -For questions about our paper or code, please contact [Daniel Bolya](mailto:dbolya@ucdavis.edu). +### YOLACTv1 ONNX runtime with TRT execution provider + +##### Prerequisites +- Pytorch (tested with 1.7.0) +- YOLACT requirements + - pip3 install -r requirements.txt +- onnxruntime with TRT execution provider + - follow detailed instructions: https://github.com/microsoft/onnxruntime/blob/master/BUILD.md#tensorrt + - one possible issue i see right away, that this build is tested with CUDA 11.0 and cuDNN 8.0. But Jetpack 4.4 supports CUDA 10.0.326 and cuDNN 7.6.3. There could be some trouble building with older versions. + - i succeded with CUDA 11.1 and cuDNN 8.0 + - add --parallel for faster build + - onnxruntime build command: ``./build.sh --parallel --cudnn_home --cuda_home --use_tensorrt --tensorrt_home `` + - if you get build error complaining about onnx-tensorrt package change ``return std::move(fields);`` to ``return fields;`` in ``builtin_op_importers.cpp`` + - python wheel build command: ``./build.sh --update --build --parallel --build_wheel --use_tensorrt --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --tensorrt_home /usr/lib/aarch64-linux-gnu`` + - don't forget to change local paths :) + - copy built python package to the yolact repo later or add ``/build/Linux/Debug/`` to your PYTHONPATH env variable + - ``export PYTHONPATH=/build/Linux/Debug:$PYTHONPATH`` + +##### Execution +- Clone this repo ``onnx`` branch + - git clone -b onnx https://github.com/simutisernestas/yolact +- Download model weights from original repo README https://github.com/dbolya/yolact + - Tested only with Resnet50-FPN (yolact_resnet50_54_800000.pth) +- Run ``export_onnx.sh`` bash script to export onnx model + - ``chmod +x export_onnx.sh`` + - ``./export_onnx.sh`` + - This will produce ``yolact.onnx`` file +- Run ``lab.py`` script (loading of model will take some time) + - ``python3 lab.py`` +- Check results directory \ No newline at end of file diff --git a/data/config.py b/data/config.py index 0e8ce72af..748ed37c5 100644 --- a/data/config.py +++ b/data/config.py @@ -513,7 +513,7 @@ def print(self): 'delayed_settings': [], # Use command-line arguments to set this. - 'no_jit': False, + 'no_jit': True, 'backbone': None, 'name': 'base_config', diff --git a/export_onnx.sh b/export_onnx.sh new file mode 100755 index 000000000..2f80a69ba --- /dev/null +++ b/export_onnx.sh @@ -0,0 +1 @@ +python3 eval_original.py --trained_model=yolact_resnet50_54_800000.pth --score_threshold=0.3 --top_k=81 --image=data/rand.jpg diff --git a/lab.py b/lab.py index 94be42dad..6448bb174 100644 --- a/lab.py +++ b/lab.py @@ -10,9 +10,14 @@ import time eval.get_args() import glob +from pathlib import Path +Path("results").mkdir(exist_ok=True) print("Loading trt execution, this may take a while... ") sess = rt.InferenceSession("yolact.onnx") +print("Onnx execution prodivers:") +print(sess.get_providers()) +print("WARNING! If above list doesn't contain TensorrtExecutionProvider model isn't executed by TensorRT") input_name = sess.get_inputs()[0].name loc_name = sess.get_outputs()[0].name conf_name = sess.get_outputs()[1].name diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 000000000..3e08ead6e --- /dev/null +++ b/requirements.txt @@ -0,0 +1,6 @@ +cython +opencv-python +pillow +pycocotools +matplotlib +numpy \ No newline at end of file