Yolox export & blade optimize support #49

wuziheng · 2022-04-29T09:09:32Z

YoloX jit export support with UT
YoloX jit export with blade optimize enable by export config. (EG:speed up yoloxs from 42fps to 123fps on 1080Ti)

Cathy0908 · 2022-05-05T04:22:51Z

easycv/toolkit/blade/cv_blade_utils.py

+        print(err)
+
+
+def blade_yolox_optimize(script_model,


blade_yolox_optimize should be as a common api，decoupling from yolox

Cathy0908 · 2022-05-05T04:24:03Z

easycv/apis/export.py

-    else:
-        print('test_pipeline not found, using default preprocessing!')
-        raise ValueError('export model config without test_pipeline')
+    cfg.model.type = 'YOLOXExport'


remove YOLOXExport api, should be a tool not a model

it may be different for different model， we use
yolox blade as name because the blacklist op for blade is specialized for yolox, whether
useful for other cnn or transformer need to be checked later

or support export mode in model

Cathy0908 · 2022-05-05T04:26:06Z

easycv/apis/export.py

+        input = 255 * torch.rand((batch_size, 3) + img_scale)
+        yolox_trace = torch.jit.trace(model_export, input.to(device))
+
+        if getattr(cfg.export, 'use_blade', False):


should be extracted to api export_blade

Cathy0908 · 2022-05-05T04:26:32Z

easycv/models/detection/yolox/yolox.py

+
+
+@MODELS.register_module
+class YOLOXExport(YOLOX):


remove YOLOXExport

using yoloXExport to fit the jit trace input/output requirenments, otherwise we should change the original YOLOX and add a config for it to open the post process.

Cathy0908 · 2022-05-05T04:27:42Z

tests/predictors/test_detector.py

@@ -39,6 +40,25 @@ def test_yolox_detector(self):
        self.assertEqual(len(output['detection_boxes']), 7)
        self.assertEqual(output['ori_img_shape'], [230, 352])

+    def test_yolox_jit_detector(self):


add unittest for export jit and blade

Cathy0908 · 2022-05-05T04:29:38Z

easycv/toolkit/blade/cv_blade_utils.py

+import torch
+import torchvision
+
+os.environ['DISC_ENABLE_STITCH'] = 'true'


os.environ['DISC_ENABLE_STITCH'] = os.environ.get('DISC_ENABLE_STITCH', 'true')
os.environ['DISC_EXPERIMENTAL_SPECULATION_TLP_ENHANCE'] = os.environ.get('DISC_EXPERIMENTAL_SPECULATION_TLP_ENHANCE', 'true')

Cathy0908 · 2022-05-05T04:31:30Z

easycv/toolkit/blade/cv_blade_utils.py

+        torch_version = 'failed'
+        torch_cuda = 'failed'
+        env_flag = False
+        print(


replace print with logging, other print too

Cathy0908 · 2022-05-05T04:33:28Z

easycv/toolkit/blade/cv_blade_utils.py

+    torch_config.customize_op_black_list = [
+        'aten::select', 'aten::index', 'aten::slice', 'aten::view'
+    ]
+    torch_config.fp16_fallback_op_ratio = 0.3


whether fp16_fallback_op_ratio support configurable?

this config is specialized for yolox, other models should be test and add configs and then we should know how to refact

done, config by outside config[export][blade_config]

Cathy0908 · 2022-05-05T04:39:37Z

easycv/toolkit/blade/cv_blade_utils.py

+            model_inputs=tuple(inputs),
+        )
+    benchmark(script_model, inputs, backend, batch, 'easycv')
+    benchmark(model, inputs, backend, batch, 'easycv script')


benchmark is not necessary for blade_yolox_optimize

use benchmark to print optimize result.

…rol blade config

…logging warning

tanyokwok · 2022-05-05T07:13:01Z

easycv/predictors/detector.py

-            det_out = self.model(
-                img, mode='test', img_metas=[data_dict['img_metas']._data])
+
+            if self.trace_able:


Nit: self.traceable

tanyokwok · 2022-05-05T07:23:33Z

easycv/toolkit/blade/cv_blade_utils.py

+os.environ['DISC_ENABLE_STITCH'] = os.environ.get('DISC_ENABLE_STITCH', 'true')
+os.environ['DISC_EXPERIMENTAL_SPECULATION_TLP_ENHANCE'] = os.environ.get('DISC_EXPERIMENTAL_SPECULATION_TLP_ENHANCE', 'true')


Those environmental variables are only needed by DISC stitch. It's recommended to be used only when the DISC stitch strategty is perfered.

tanyokwok · 2022-05-05T07:23:54Z

easycv/toolkit/blade/cv_blade_utils.py

+    import ctypes
+    _cudart = ctypes.CDLL('libcudart.so')


Nit: not needed here.

will cause a import error even we dont use it.

tanyokwok · 2022-05-05T07:24:45Z

easycv/toolkit/blade/cv_blade_utils.py

+
+
+@contextmanager
+# def opt_trt_config(enable_fp16=True):


Nit: remove this line

tanyokwok · 2022-05-05T07:26:32Z

easycv/toolkit/blade/cv_blade_utils.py

+@contextmanager
+def opt_blade_mixprec():
+    try:
+        dummy = torch.classes.torch_blade.MixPrecision(True)
+        yield
+    finally:
+        pass
+
+
+@contextmanager
+def opt_disc_config(enable_fp16=True):
+    torch_config = torch_blade.config.Config()
+    torch_config.enable_fp16 = enable_fp16
+    try:
+        with torch_config:
+            yield
+    finally:
+        pass
+


Nit: Add those configurations if needed in the future. I prefer to remove those configurations.

tanyokwok · 2022-05-05T07:30:38Z

easycv/toolkit/blade/cv_blade_utils.py

+results = []
+
+
+def printStats(backend, timings, batch_size=1, model_name='default'):


It seems that the following codes have no relation to Blade but are related to the benchmark. I prefer to add a benchmark tutorial in some place.

…sor during jit trace and failed

Cathy0908

revert approve

wenmengzhou · 2022-05-18T07:10:31Z

refactor this CR in #66

wuziheng · 2022-05-18T07:38:47Z

LGTM

周楼 added 4 commits April 29, 2022 11:27

Yolox JIT export & YoloXPredictor

ab26049

YoloX export with blade optimize support with config

3340067

Add Yolox preidctor UT

c6669fe

Fix pre-commit bug

518cc29

wenmengzhou added the enhancement New feature or request label Apr 30, 2022

Cathy0908 approved these changes May 5, 2022

View reviewed changes

周楼 added 6 commits May 5, 2022 14:11

fix use_blade to export_blade

57c73af

fix cv_blade_utils env with get default

01cd311

replace print with logging

237d8d8

replace yolox_blade_optimize to blade_optimize and add config to cont…

7d99c93

…rol blade config

add blade_config default setting with logging info, user config with …

b7a727a

…logging warning

pre-commit fix

2cb69cd

tanyokwok reviewed May 5, 2022

View reviewed changes

周楼 added 4 commits May 5, 2022 15:51

Add yolox export UT and yolox jit export UT

26d94db

pre commit fix

db4928f

add failback setting to solve some detection model generate empty ten…

c268c56

…sor during jit trace and failed

replace trace_able with traceable

c4f232c

Cathy0908 reviewed May 5, 2022

View reviewed changes

Cathy0908 self-requested a review May 5, 2022 12:19

wenmengzhou closed this May 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yolox export & blade optimize support #49

Yolox export & blade optimize support #49

wuziheng commented Apr 29, 2022

Cathy0908 May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 6, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

wuziheng May 5, 2022

Cathy0908 May 5, 2022

wuziheng May 5, 2022

wuziheng May 5, 2022

tanyokwok May 5, 2022

tanyokwok May 5, 2022

tanyokwok May 5, 2022

wuziheng May 6, 2022

tanyokwok May 5, 2022

tanyokwok May 5, 2022

wuziheng May 6, 2022

tanyokwok May 5, 2022

Cathy0908 left a comment

wenmengzhou commented May 18, 2022

wuziheng commented May 18, 2022

		os.environ['DISC_ENABLE_STITCH'] = os.environ.get('DISC_ENABLE_STITCH', 'true')
		os.environ['DISC_EXPERIMENTAL_SPECULATION_TLP_ENHANCE'] = os.environ.get('DISC_EXPERIMENTAL_SPECULATION_TLP_ENHANCE', 'true')

		results = []


		def printStats(backend, timings, batch_size=1, model_name='default'):



		@MODELS.register_module
		class YOLOXExport(YOLOX):

Yolox export & blade optimize support #49

Yolox export & blade optimize support #49

Conversation

wuziheng commented Apr 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cathy0908 left a comment

Choose a reason for hiding this comment

wenmengzhou commented May 18, 2022

wuziheng commented May 18, 2022