Merge pull request openvinotoolkit#827 from IRDonch/release-2020.1

Merge OpenVINO toolkit 2020.1 content into master

Author: Roman Donchenko

.gitattributes

@@ -7,3 +7,18 @@
 /demos/build_demos_msvc.bat omz.package=w
 /demos/tests/** -omz.package
 /tools/downloader/tests/** -omz.package
+
+/ci/requirements-ac.txt omz.ci.job-for-change.ac
+/ci/requirements-conversion.txt omz.ci.job-for-change.demos omz.ci.job-for-change.models
+/ci/requirements-demos.txt omz.ci.job-for-change.demos
+/ci/requirements-downloader.txt omz.ci.job-for-change.demos omz.ci.job-for-change.models
+
+/demos/** omz.ci.job-for-change.demos
+/demos/**/*.md -omz.ci.job-for-change.demos
+
+/models/** omz.ci.job-for-change.documentation
+/models/**/*.yml -omz.ci.job-for-change.documentation
+
+/tools/accuracy_checker/** omz.ci.job-for-change.ac
+/tools/downloader/** omz.ci.job-for-change.downloader
+/tools/**/*.md -omz.ci.job-for-change.ac -omz.ci.job-for-change.downloader

CONTRIBUTING.md

@@ -0,0 +1,276 @@
+# How to Contribute Models to Open Model Zoo
+
+We appreciate your intention to contribute model to the OpenVINO™ Open Model Zoo (OMZ). OMZ is licensed under the Apache\* License, Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms. Note that we accept models under permissive licenses, such as **MIT**, **Apache 2.0**, and **BSD-3-Clause**. Otherwise, it might take longer time to get your model approved.
+
+Frameworks supported by the Open Model Zoo: 
+* Caffe\*
+* Caffe2\* (via conversion to ONNX\*)
+* TensorFlow\*
+* PyTorch\* (via conversion to ONNX\*)
+* MXNet\*
+
+Open Model Zoo also supports models already in the ONNX format.
+
+## Pull Request Requirements
+
+To contribute to OMZ, create a pull request (PR) in this repository using the `develop` branch.
+Pull requests are strictly formalized and are reviewed by the OMZ maintainers for consistence and legal compliance.
+
+Each PR contributing a model must contain:
+* [configuration file `model.yml`](#configuration-file)
+* [documentation of model in markdown format](#documentation)
+* [accuracy validation configuration file](#accuracy-validation)
+* (*optional*) [demo](#demo)
+
+Follow the rules in the sections below before submitting a pull request.
+
+### Model Name
+
+Name your model in OMZ according to the following rules:
+- Use a name that is consistent with an original name, but complete match is not necessary
+- Use lowercase
+- Use `-`(preferable) or `_` as delimiters, for spaces are not allowed
+- Add a suffix according to framework identifier (see **`framework`** description in the [configuration file](#configuration-file) section for examples), if the model is a reimplementation of an existing model from another framework
+
+This name will be used for downloading, converting, and other operations.
+Examples of model names:
+- `resnet-50-pytorch`
+- `mobilenet-v2-1.0-224`
+
+### Files Location
+
+Place your files as shown in the table below:
+
+File | Destination
+---|---
+configuration file | `models/public/<model_name>/model.yml`
+documentation file | `models/public/<model_name>/<model_name>.md`
+validation configuration file|`tools/accuracy_checker/configs/<model_name>.yml`
+demo|`demos/<demo_name>`<br>or<br>`demos/python_demos/<demo_name>`
+
+### Tests
+
+Your PR must pass next tests:
+* Model is downloadable by the `tools/downloader/downloader.py` script. See [Configuration file](#configuration-file) for details.
+* Model is convertible by the `tools/downloader/converter.py` script. See [Model conversion](#model-conversion) for details.
+* Model is usable by demo or sample and provides adequate results. See [Demo](#demo) for details.
+* Model passes accuracy validation. See [Accuracy validation](#accuracy-validation) for details.
+
+
+### PR Rejection
+
+Your PR may be rejected in some cases, for example:
+* If a license is inappropriate (such as GPL-like licenses).
+* If a dataset is inaccessible.
+* If the PR fails one of the tests above.
+
+## Configuration File
+
+The model configuration file contains information about model: what it is, how to download it, and how to convert it to the IR format. This information must be specified in the `model.yml` file that must be located in the model subfolder. 
+
+The detailed descriptions of file entries provided below.
+
+**`description`**
+
+Description of the model. Must match with the description from the model [documentation](#documentation).
+
+**`task_type`**
+
+[Model task type](tools/downloader/README.md#model-information-dumper-usage). If there is no task type of your model, add a new one to the list `KNOWN_TASK_TYPES` of the [tools/downloader/common.py](tools/downloader/common.py) file.
+
+**`files`**
+
+> **NOTE**: Before filling this section, make sure that the model can be downloaded either via a direct HTTP(S) link or from Google Drive\*.
+
+Downloadable files. Each file is described by:
+
+* `name` - sets a file name after downloading
+* `size` - sets a file size
+* `sha256`  - sets a file hash sum
+* `source` - sets a direct link to a file *OR* describes a file access parameters
+
+> **TIP**: You can obtain a hash sum using the `sha256sum <file_name>` command on Linux\*.
+ 
+If file is located on Google Drive\*, the `source` section must contain:
+- `$type: google_drive`
+- `id` file ID on Google Drive\*
+
+> **NOTE:** If file is on GitHub\*, use the specific file version.
+
+**`postprocessing`** (*optional*)
+
+Post processing of the downloaded files.
+
+For unpacking archive:
+- `$type: unpack_archive`
+- `file` — Archive file name
+- `format` — Archive format (zip | tar | gztar | bztar | xztar)
+
+For replacement operation:
+- `$type: regex_replace`
+- `file` — Name of file to run replacement in
+- `pattern` — [Regular expression](https://docs.python.org/3/library/re.html)
+- `replacement` — Replacement string
+- `count` (*optional*)  — Exact number of replacements (if number of `pattern` occurrences less then this number, downloading will be aborted)
+
+**`conversion_to_onnx_args`** (*only for Caffe2\*, PyTorch\* models*)
+
+List of ONNX\* conversion parameters, see `model_optimizer_args` for details.
+
+**`model_optimizer_args`**
+
+Conversion parameters (learn more in the [Model conversion](#model-conversion) section). For example:
+```
+  - --input=data
+  - --mean_values=data[127.5]
+  - --scale_values=data[127.5]
+  - --reverse_input_channels
+  - --output=prob
+  - --input_model=$conv_dir/googlenet-v3.onnx
+```
+
+> **NOTE:** Do not specify `framework`, `data_type`, `model_name` and `output_dir`, since they are deduced automatically.
+
+> **NOTE:** `$dl_dir` used to substitute subdirectory of downloaded model and `$conv_dir` used to substitute subdirectory of converted model.
+
+**`framework`**
+
+Framework of the original model (see [here](tools/downloader/README.md#model-information-dumper-usage) for details).
+
+**`license`**
+
+URL of the model license.
+
+### Example
+
+This example shows how to download the classification model [DenseNet-121*](models/public/densenet-121-tf/model.yml) pretrained in TensorFlow\* from Google Drive\* as an archive.
+
+```
+description: >-
+  This is a TensorFlow\* version of `densenet-121` model, one of the DenseNet
+  group of models designed to perform image classification. The weights were converted
+  from DenseNet-Keras Models. For details see repository <https://github.com/pudae/tensorflow-densenet/>,
+  paper <https://arxiv.org/pdf/1608.06993.pdf>
+task_type: classification
+files:
+  - name: tf-densenet121.tar.gz
+    size: 30597420
+    sha256: b31ec840358f1d20e1c6364d05ce463cb0bc0480042e663ad54547189501852d
+    source:
+      $type: google_drive
+      id: 0B_fUSpodN0t0eW1sVk1aeWREaDA
+postprocessing:
+  - $type: unpack_archive
+    format: gztar
+    file: tf-densenet121.tar.gz
+model_optimizer_args:
+  - --reverse_input_channels
+  - --input_shape=[1,224,224,3]
+  - --input=Placeholder
+  - --mean_values=Placeholder[123.68,116.78,103.94]
+  - --scale_values=Placeholder[58.8235294117647]
+  - --output=densenet121/predictions/Reshape_1
+  - --input_meta_graph=$dl_dir/tf-densenet121.ckpt.meta
+framework: tf
+license: https://raw.githubusercontent.com/pudae/tensorflow-densenet/master/LICENSE
+```
+
+## Model Conversion
+
+Deep Learning Inference Engine (IE) supports models in the Intermediate Representation (IR) format. A model from any supported framework can be converted to IR using the Model Optimizer tool included in the OpenVINO&trade; toolkit. Find more information about conversion in the [Model Optimizer Developer Guide](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html). After a successful conversion you get a model in the IR format, with the `*.xml` file representing the net graph and the `*.bin` file containing the net parameters. 
+
+> **NOTE 1**: Image preprocessing parameters (mean and scale) must be built into a converted model to simplify model usage.
+
+> **NOTE 2**: If a model input is a color image, color channel order should be `BGR`.
+
+## Demo
+
+A demo shows the main idea of how to infer a model using IE. If your model solves one of the tasks supported by the Open Model Zoo, try to find an appropriate option from [demos](demos/README.md) or [samples](https://docs.openvinotoolkit.org/latest/_docs_IE_DG_Samples_Overview.html). Otherwise, you must provide your own demo (C++ or Python). 
+
+The demo's name should end with `_demo` suffix to follow the convention of the project.
+
+Demos are required to support the following keys:
+
+ -  `-i "<input>"`: Required. Input to process.
+ -  `-m "<path>"`: Required. Path to an .xml file with a trained model. If the demo uses several models at the same time, use other keys prefixed with `-m_`.
+ -  `-d "<device>"`: Optional. Specifies a target device to infer on. CPU, GPU, FPGA, HDDL or MYRIAD is acceptable. Default must be CPU. If the demo uses several models at the same time, use keys prefixed with `d_` (just like keys `m_*` above) to specify device for each model.
+ -  `-no_show`: Optional. Do not visualize inference results.
+
+> **TIP**: For Python, it is preferable to use `--` instead of `-` for long keys.
+
+You can also add any other necessary parameters.
+
+Add `README.md` file, which describes demo usage. Update [demos' README.md](demos/README.md) adding your demo to the list.
+
+## Accuracy Validation
+
+Accuracy validation can be performed by the [Accuracy Checker](./tools/accuracy_checker) tool. This tool can use either IE to run a converted model, or an original framework to run an original model. Accuracy Checker supports lots of datasets, metrics and preprocessing options, which simplifies validation if a task is supported by the tool. You only need to create a configuration file that contains necessary parameters for accuracy validation (specify a dataset and annotation, pre- and post-processing parameters, accuracy metrics to compute and so on) of converted model. For details, refer to [Testing new models](./tools/accuracy_checker#testing-new-models).
+
+If a model uses a dataset which is not supported by the Accuracy Checker, you also must provide the license and the link to it and mention it in the PR description. 
+
+When the configuration file is ready, you must run the Accuracy Checker to obtain metric results. If they match your results, that means conversion was successful and the Accuracy Checker fully supports your model, metric and dataset. Otherwise, recheck the [conversion](#model-conversion) parameters or the validation configuration file.
+
+### Example
+
+This example uses one of the files from `tools/accuracy_checker/configs` — validation configuration file for [DenseNet-121](tools/accuracy_checker/configs/densenet-121-tf.yml)\* from TensorFlow\*:
+```
+models:
+  - name: densenet-121-tf
+    launchers:
+      - framework: dlsdk
+        tags:
+          - FP32
+        model:   public/densenet-121-tf/FP32/densenet-121-tf.xml
+        weights: public/densenet-121-tf/FP32/densenet-121-tf.bin
+        adapter: classification
+
+      - framework: dlsdk
+        tags:
+          - FP16
+        model:   public/densenet-121-tf/FP16/densenet-121-tf.xml
+        weights: public/densenet-121-tf/FP16/densenet-121-tf.bin
+        adapter: classification
+
+    datasets:
+      - name: imagenet_1000_classes
+        preprocessing:
+          - type: resize
+            size: 256
+          - type: crop
+            size: 224
+```
+
+
+## Documentation
+
+Documentation is a very important part of model contribution as it helps to better understand the possible usage of the model. Documentation must be named in accordance with the name of the model.
+The documentation should contain:
+* description of a model
+	* main purpose
+	* features
+	* references to a paper or/and a source
+* model specification
+	* type
+	* framework
+	* GFLOPs (*if available*)
+	* number of parameters (*if available*)
+* validation dataset description and/or a link
+* main accuracy values (also description of a metric)
+* detailed description of input and output for original and converted models
+* the model's licensing terms
+
+Learn the detailed structure and headers naming convention from any model documentation (for example, [alexnet](./models/public/alexnet/alexnet.md)).
+
+## Legal Information
+
+[\*] Other names and brands may be claimed as the property of others.
+
+OpenVINO is a trademark of Intel Corporation or its subsidiaries in the U.S. and/or other countries.
+
+Copyright &copy; 2018-2019 Intel Corporation
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+```
+http://www.apache.org/licenses/LICENSE-2.0
+```
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

README.md

@@ -1,6 +1,5 @@
 # [OpenVINO™ Toolkit](https://01.org/openvinotoolkit) - Open Model Zoo repository
-[![Build Status](http://134.191.240.124/buildStatus/icon?job=omz/2018/trigger)](http://134.191.240.124/job/omz/job/2018/job/trigger/)
-[![Stable release](https://img.shields.io/badge/version-2019_R3.1-green.svg)](https://github.com/opencv/open_model_zoo/releases/tag/2019_R3.1)
+[![Stable release](https://img.shields.io/badge/version-2020.1-green.svg)](https://github.com/opencv/open_model_zoo/releases/tag/2020.1)
 [![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/open_model_zoo/community)
 [![Apache License Version 2.0](https://img.shields.io/badge/license-Apache_2.0-green.svg)](LICENSE)
 
@@ -34,6 +33,8 @@ We welcome community contributions to the Open Model Zoo repository. If you have
 * In case of a larger feature, provide a relevant demo.
 * Submit a pull request at https://github.com/opencv/open_model_zoo/pulls
 
+You can find additional information about model contribution [here](CONTRIBUTING.md).
+
 We will review your contribution and, if any additional fixes or modifications are needed, may give you feedback to guide you. When accepted, your pull request will be merged into the GitHub* repositories.
 
 Open Model Zoo is licensed under Apache License, Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.

ci/get-jobs-for-changes.py

@@ -0,0 +1,72 @@
+#!/usr/bin/python3
+
+# Copyright (c) 2019 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+A script that prints a JSON description of the CI jobs necessary to validate
+the changes made between a base commit and the current commit.
+
+The output format is a an object where each key is the identifier of the job
+and the corresponding value represents that job's parameters. The value
+is usually just "true" (which just means that the job should be run), but
+for the "models" job the value can be an array of names of models that should
+be validated.
+"""
+
+import argparse
+import json
+import re
+import subprocess
+import sys
+
+from pathlib import Path, PurePosixPath
+
+RE_ATTRIB_NAME = re.compile(r"omz\.ci\.job-for-change\.(.+)")
+
+def group_by_n(iterable, n):
+    return zip(*[iter(iterable)] * n)
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('base_commit', metavar='COMMIT')
+    args = parser.parse_args()
+
+    git_diff_output = subprocess.check_output(
+        ["git", "diff", "--name-only", "-z", args.base_commit + "...HEAD"])
+    changed_files = list(map(PurePosixPath, git_diff_output.decode()[:-1].split("\0")))
+
+    models_dir = PurePosixPath("models")
+
+    jobs = {}
+
+    for changed_file in changed_files:
+        if models_dir in changed_file.parents and changed_file.name == "model.yml":
+            if Path(changed_file).exists(): # it might've been deleted in the branch
+                jobs.setdefault("models", []).append(changed_file.parent.name)
+
+    git_check_attr_output = subprocess.run(
+        ["git", "check-attr", "--stdin", "-z", "--all"],
+        input=git_diff_output, stdout=subprocess.PIPE, check=True).stdout
+
+    for path, attribute, value in group_by_n(git_check_attr_output.decode()[:-1].split("\0"), 3):
+        attribute_match = RE_ATTRIB_NAME.fullmatch(attribute)
+        if value != 'unset' and attribute_match:
+            jobs[attribute_match.group(1)] = True
+
+    json.dump(jobs, sys.stdout)
+    print()
+
+if __name__ == "__main__":
+    main()