Added script to test caffe converted models layer by layer

[arikpoz]

I've added a script which compares layer by layer both the weights and the outputs of a caffe model and its converted mxnet model.

The script by default runs on the known test models mentioned in test_converter.py, namely: bvlc_googlenet, vgg-16 and resnet-50.
In addition the script offers functions for comparing any model by supplying the parameters of the caffe model.

The script goes over the layers of the network DAG, in BFS order, and compares the layer's parameters, usually weights and bias, but others as well.

In addition the script takes an image as input and run inference using both caffe and mxnet models, and compare the blob outputs of each layer, taking into consideration inplace layers.

For each weight blob and output blob the script log to screen the mean and max of the absolute difference between the blobs.

Attached output generated when running the script using default parameters: compare_layers_output.txt

Since the script runs caffe inference, it depends on caffe being installed. The dependency is implicit, so the user of the script can resolve it by installing his favorite caffe version and setting the proper environment variables. All the script requires is that 'import caffe' will work.

[arikpoz]

Note that this PR addresses the 'pair test with caffe' in future roadmap of #5247

[mli]

Can you please add a CI test on https://github.com/dmlc/mxnet/blob/master/Jenkinsfile#L335

[arikpoz]

I'm not sure how to go about that..
Is running the tool enough? should I add assertion to make sure the per layer difference is low?
Also, the tool depends on caffe being setup properly (such that 'import caffe' works), can this be an issue for integration tests?

[mli]

the ci docker has caffe installed, also gpu is enabled if you want to use. see. https://github.com/dmlc/mxnet/blob/master/tests/ci_build/Dockerfile.caffe_gpu i suggest to download a model, feed in an image, and compared the results to make sure that the difference is small the current test script is here: https://github.com/dmlc/mxnet/blob/master/tools/caffe_converter/test_converter.py

…

On Tue, Jun 13, 2017 at 11:29 AM, Arik Poznanski ***@***.***> wrote: I'm not sure how to go about that.. Is running the tool enough? should I add assertion to make sure the per layer difference is low? Also, the tool depends on caffe being setup properly (such that 'import caffe' works), can this be an issue for integration tests? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6680 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZv4aVKrxtVDxXCl_g8eRKQ3KWXSg28ks5sDtUKgaJpZM4N4Atd> .

[arikpoz]

Currently test_converter.py downloads the models and dataset and check the models performance,
while compare_layers.py downloads the models and a test image and check the models weights and outputs.

Do you think the test_converter.py script should cover both checks? in which case I need to merge them to a single file.

Or should I leave compare_layers.py as a standalone script, add the relevant assertions, possibly rename it to test_compare_layers.py, and add it to the list of CI tests?

[mli]

i prefer to add the test codes into test_converter.py. but i'm ok with the second option as well

…

On Tue, Jun 13, 2017 at 11:53 AM, Arik Poznanski ***@***.***> wrote: Currently *test_converter.py* downloads the models and dataset and check the models performance, while *compare_layers.py* downloads the models and a test image and check the models weights and outputs. Do you think the *test_converter.py* script should cover both checks? in which case I need to merge them to a single file. Or should I leave *compare_layers.py* as a standalone script, add the relevant assertions, possibly rename it to *test_compare_layers.py*, and add it to the list of CI tests? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6680 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZv4Zz3Re4CzgP8j5oEUyopO4SyqNszks5sDtqygaJpZM4N4Atd> .

[arikpoz]

I've refactored test_converter.py script to run both performance test and layer-by-layer comparison test for known models. The layer comparison login remains in a standalone compare_layers.py script, to avoid clouding the test_converter script, and to allow users to run the layer by layer comparison on their private models.

I've added assertion of mean difference allowed (1e-4) and max difference allowed (1e-2), based on actual numbers I got when running on CPU. Unfortunately, I can't run the test on GPU, so I hope it passes the integration test, and if not, we might need to update the threshold for the GPU. Hopefully, there won't be any significant difference..

[arikpoz]

Finally, continuous-integration/jenkins/pr-head passed with the new test.
I'm not familiar with the problem of continuous-integration/travis-ci/pr, if someone can check it out it would be great..

[arikpoz]

ready to merge

[arikpoz]

@mli can you check?

[arikpoz]

errors doesn't seem to be related