Add ruletest builder and profiling backend

docs/backend.md

@@ -0,0 +1,73 @@
+# Setup Device and Backend
+
+To run a model on the device, we implement several backends. These backends will run the model and parse the command line outputs to get latency results. We provide a consistent API for such backends. You can inherit the `BaseBackend` and implement a backend for your own device that's not included here.
+
+Currently we support TFLite on CPU, GPU and OpenVINO on VPU. Following we will explain how to setup the device and backend.
+
+## Android Guide
+
+### Prepare Android Device
+
+> Follow [ADB Guide](https://developer.android.com/studio/command-line/adb) to install adb on your host device.
+
+> Follow the [tensorflow official guide](https://www.tensorflow.org/lite/performance/measurement) for a more detailed guide to build and deploy `benchmark_model` onto the device.
+
+
+Download tensorflow source code and type following lines in tensorflow root folder:
+```
+bazel build -c opt //tensorflow/lite/tools/benchmark:benchmark_model
+adb push bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model /data/local/tmp
+adb shell chmod +x /data/local/tmp/benchmark_model
+```
+
+### Usage
+
+Use api `get_backend` to initialize the Android CPU or GPU backend.
+
+```python
+backend = get_backend('tflite_cpu', {
+    'MODEL_DIR': '~/models',  # directory on host to save temporary tflite models
+    'REMOTE_MODEL_DIR': '/data/local/tmp/models',  # directory on mobile phone to place models
+    'KERNEL_PATH': '/data/local/tmp/kernels.cl',  # directory on mobile phone where kernel code files will be generated
+    'BENCHMARK_MODEL_PATH': '/data/local/tmp/benchmark_model',  # path to bin of `benchmark_model`
+    'DEVICE_SERIAL': '',  # serial id of the device. set to '' if there is only one device connected to your host
+})
+```
+
+### Modified Build for TFLite GPU
+
+TODO
+
+## VPU Guide
+
+### Prerequisits
+
+#### OpenVINO
+
+Follow [OpenVINO Installation Guide](https://docs.openvinotoolkit.org/latest/installation_guides.html) to install openvino on your host.
+
+#### Python Environments
+
+Because some tools for VPU use a different tensorflow and python version from nn_meter, so you need to provide seperate environments for the main tool and VPU. We recommend to use [virtualenv](https://virtualenv.pypa.io/en/latest/). We use python3.6 as our test enviroment.
+
+```
+virtualenv openvino_env
+source openvino_env/bin/activate
+pip install -r openvino_env.txt
+deactivate
+```
+
+### Usage
+
+Suppose you place the python environments at `~/openvino_env`.
+
+```python
+backend = get_backend('vpu', {
+    'OPENVINO_ENV': '~/openvino_env',
+    'OPTIMIZER_PATH': '/data/openvino_2019.2.242/deployment_tools/model_optimizer/mo_tf.py',
+    'TMP_DIR': '~/models',
+    'OPENVINO_RUNTIME_DIR': '/data/openvino_2019.2.242/bin',
+    'DEVICE_SERIAL': '/dev/ttyUSB4',
+    'DATA_TYPE': 'FP16',
+})
+```

docs/openvino_requirements.txt

@@ -0,0 +1,6 @@
+tensorflow>=1.2.0,<2.0.0
+networkx==2.3.0
+numpy>=1.12.0
+test-generator==0.1.1
+defusedxml>=0.5.0
+serial

docs/rule_builder.md

@@ -0,0 +1,141 @@
+# RuleTest Builder
+
+This part generates and profiles models (what we call "test cases") on a given device to get rules.
+
+## End-to-End Demo
+
+```python
+# initialize backend
+backend = get_backend(
+    backend = 'tflite_cpu', 
+    params = {
+        'MODEL_DIR': '/data/jiahang/test_models',
+        'REMOTE_MODEL_DIR': '/mnt/sdcard/tflite_bench',
+        'KERNEL_PATH': '/mnt/sdcard/tflite_bench/kernel.cl',
+        'BENCHMARK_MODEL_PATH': '/data/local/tmp/benchmark_model_fixed_group_size',
+        'DEVICE_SERIAL': '0000028e2c780e4e',
+    }
+)
+
+# generate testcases
+testcases = get_testcases(model_dir='/data/jiahang/test_models')
+
+# run testcases and collect profiling results
+profile_results = run_testcases(backend, testcases)
+
+# determine fusion rules from profiling results
+detect_results = get_fusionrule(profile_results)
+```
+
+`backend` refers to device and framework to execute the model. Currently we support TFLite on CPU, GPU and OpenVINO on VPU. Refer to [backend guide](./backend.md) for how to setup the device and backend.
+
+Note that we provide a consistent API for backend, so you can inherit the `BaseBackend` and implement a backend for your own device that's not included here. 
+
+Also note that it's optional to use a backend. What `run_testcases` do is just collecting latency results of each testcases, so you can use your own tools to measure the latency. Refer to implementation of `run_testcases` for how to fill back the latency.
+
+## Data Structure of TestCases
+
+This is a json dump of generated testcases.
+
+```json
+{
+    "BF_reshape_reshape": {
+        "reshape_1": {
+            "model": "/Users/kalineid/test_models/BF_reshape_reshape_reshape_1",
+            "shapes": [
+                [
+                    28,
+                    28,
+                    16
+                ]
+            ]
+        },
+        "reshape_2": {
+            "model": "/Users/kalineid/test_models/BF_reshape_reshape_reshape_2",
+            "shapes": [
+                [
+                    16,
+                    28,
+                    28
+                ]
+            ]
+        },
+        "block": {
+            "model": "/Users/kalineid/test_models/BF_reshape_reshape_block",
+            "shapes": [
+                [
+                    28,
+                    28,
+                    16
+                ]
+            ]
+        }
+    },
+    "BF_reshape_dwconv": {
+        "reshape": {
+            "model": "/Users/kalineid/test_models/BF_reshape_dwconv_reshape",
+            "shapes": [
+                [
+                    28,
+                    28,
+                    16
+                ]
+            ]
+        },
+        "dwconv": {
+            "model": "/Users/kalineid/test_models/BF_reshape_dwconv_dwconv",
+            "shapes": [
+                [
+                    16,
+                    28,
+                    28
+                ]
+            ]
+        },
+        "block": {
+            "model": "/Users/kalineid/test_models/BF_reshape_dwconv_block",
+            "shapes": [
+                [
+                    28,
+                    28,
+                    16
+                ]
+            ]
+        }
+    },
+    "BF_reshape_relu": {
+        "reshape": {
+            "model": "/Users/kalineid/test_models/BF_reshape_relu_reshape",
+            "shapes": [
+                [
+                    28,
+                    28,
+                    16
+                ]
+            ]
+        },
+        "relu": {
+            "model": "/Users/kalineid/test_models/BF_reshape_relu_relu",
+            "shapes": [
+                [
+                    16,
+                    28,
+                    28
+                ]
+            ]
+        },
+        "block": {
+            "model": "/Users/kalineid/test_models/BF_reshape_relu_block",
+            "shapes": [
+                [
+                    28,
+                    28,
+                    16
+                ]
+            ]
+        }
+    }
+}
+```
+
+`BF_reshape_dwconv` is the name of a rule. It consists of several test models to profile. Here, there are three models called `reshape`, `dwconv` and `block`. For each model, the `model` field is the path to where the model is saved. `shapes` is its inputs. For example, here `[[28, 28, 16]]` means this model has only one input, and the shape is `(28, 28, 16)`.

examples/local_config_example.py

@@ -0,0 +1,48 @@
+import os
+
+ENABLED = ['MON']
+
+HOME_DIR = os.path.expanduser('~')
+
+BASE_DIR = os.path.dirname(__file__)
+
+BACKENDS = {
+    'tflite_gpu': {
+        'ENGINE': 'backends.tflite_gpu',
+        'PARAMS': {
+            'MODEL_DIR': '/data1/datasets_pad', #os.path.join(HOME_DIR, "benchmarks/models/tflite"),
+            'REMOTE_MODEL_DIR': '/mnt/sdcard/tflite_bench',
+            'KERNEL_PATH': '/mnt/sdcard/tflite_bench/kernel.cl',
+            'BENCHMARK_MODEL_PATH': '/data/local/tmp/benchmark_model_fixed_group_size',
+            'DEVICE_SERIAL': '5e6fecf',
+        },
+        'ENABLED': True,
+    },
+    'tflite_cpu': {
+        'ENGINE': 'backends.tflite_cpu',
+        'PARAMS': {
+            'MODEL_DIR': os.path.join(HOME_DIR, "benchmarks/models/tflite"),
+            'REMOTE_MODEL_DIR': '/mnt/sdcard/tflite_bench',
+            'KERNEL_PATH': '/mnt/sdcard/tflite_bench/kernel.cl',
+            'BENCHMARK_MODEL_PATH': '/data/local/tmp/benchmark_model_fixed_group_size',
+            'DEVICE_SERIAL': '5e6fecf',
+        },
+        'ENABLED': True,
+    },
+    'vpu': {
+        'ENGINE': 'backends.vpu',
+        'PARAMS': {
+            'OPENVINO_ENV': os.path.join(BASE_DIR, 'openvino_env'),
+            'OPTIMIZER_PATH': '/data/openvino_2019.2.242/deployment_tools/model_optimizer/mo_tf.py',
+            'TMP_DIR': os.path.join(HOME_DIR, 'benchmarks/openvino'),
+            'OPENVINO_RUNTIME_DIR': '/data/openvino_2019.2.242/bin',
+            'DEVICE_SERIAL': '/dev/ttyUSB4',
+            'DATA_TYPE': 'FP16',
+        },
+        'ENABLED': False,
+    }
+}
+
+OUTPUT_PATH = './gpuconv-bn-relu-test_tflite_gpu.csv'
+
+DETAIL = True