Caffe converter test fails, causing CI to halt for all PRs

[zhreshold]

Steps to reproduce

or if you are running standard examples, please provide the commands you have run that lead to the error.

1. python tools/caffe_converter/test_converter.py

What have you tried to solve it?

Tried to debug the metric_update process, figured out for vgg-16 and resnet, the network prediction output is wrong, we have the last conv output mixed to the final prediction. Thus causing the mismatched shape with label.

('!!!labels', 1, (32L,))
('!!!preds', 2, (32L, 1000L), (32L, 512L, 7L, 7L))

For googlenet, the output is correct:

('!!!labels', 1, (32L,))
('!!!preds', 2, (32L, 1000L))

@joey2014
https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/master/152/pipeline/

INFO:root:Saved checkpoint to "./model/vgg-16-0000.params"
data/val-5k-256.rec
[18:21:16] src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: data/val-5k-256.rec, use 4 threads for decoding..
('!!!labels', 1, (32L,))
('!!!preds', 2, (32L, 1000L), (32L, 512L, 7L, 7L))
Traceback (most recent call last):
  File "test_converter.py", line 89, in <module>
    main()
  File "test_converter.py", line 86, in main
    test_imagenet_model_performance(m, val, gpus, batch_size)
  File "test_converter.py", line 38, in test_imagenet_model_performance
    **mean_args)
  File "/home/ubuntu/debug/incubator-mxnet/tools/caffe_converter/../../example/image-classification/score.py", line 60, in score
    mod.update_metric(m, batch.label)
  File "/home/ubuntu/verify/mxnet/python/mxnet/module/module.py", line 718, in update_metric
    self._exec_group.update_metric(eval_metric, labels)
  File "/home/ubuntu/verify/mxnet/python/mxnet/module/executor_group.py", line 565, in update_metric
    eval_metric.update_dict(labels_, preds)
  File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 91, in update_dict
    self.update(label, pred)
  File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 373, in update
    check_label_shapes(labels, preds)
  File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 24, in check_label_shapes
    "predictions {}".format(label_shape, pred_shape))
ValueError: Shape of labels 1 does not match shape of predictions 2

[joey2014]

The layer input name in some cases is changed which causes the layer added to group by mistake, for example, in vgg16, the layer fc6 input name pool5 is changed to flatten_0. Moving the state recording code to the begin of the function can fix the issue. detail see the fix .