TF's COCO evaluation wrapper doesn't actually support include_metrics_per_category and all_metrics_per_category

[netanel-s]

System information

• What is the top-level directory of the model you are using: models/research/object_detection/
• Have I written custom code: stock script of COCO's detection evaluation
• OS Platform and Distribution: Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.8.0
• Bazel version (if compiling from source): -
• CUDA/cuDNN version: CUDA 9
• GPU model and memory: Titan XP, 12GB
• Exact command to reproduce: object_detection/eval.py with checkpoint and config file of ssdlite_mobilenet_v2_coco_2018_05_09 with added metrics_set: "coco_detection_metrics" and include_metrics_per_category: true in eval_config.

Describe the problem

Flags include_metrics_per_category and all_metrics_per_category in eval_config of the config file of the model are not used, and therefore are always False.
Hence, the output metrics are always the usual 12 metrics of COCO's AP and AR for all categories together.

Source code / logs

In get_evaluators() of object_detection/evaluator.py:
EVAL_METRICS_CLASS_DICT[eval_metric_fn_key](categories=categories)
should also be using include_metrics_per_category and all_metrics_per_category in case they're existing attributes of eval_config.
But then I get the 'Category stats do not exist' in ComputeMetrics() of object_detection/metrics/coco_tools.py since COCOEvalWrapper instance doesn't have category_stats attribute.
I tried figuring out how these are supposed to be calculated and fed, but I'm not sure the code supports per category stats (while ComputeMetrics() suggests that it is).

A fix would be highly appreciated.
Thank you very much in advance.

[netanel-s]

Just to make sure we're on the same page, the OD API has changed, but this part is essentially the same.
When include_metrics_per_category is set to be true, I get the same 'Category stats do not exist' in ComputeMetrics().
And why all_metrics_per_category is no longer a flag for eval_config?

Thanks in advance!

[bsaendig]

+1
I am experiencing the same error. Changing the include_metrics_per_category parameter to true leads to ValueError: Category stats do not exist.
A fix would be very helpful.

I looked at the files. So as i understand it, in object_detection/metrics/coco_tools.py there is a wrapper for COCOeval (from pycocotools) which somehow expects that class to hold a dictionary 'category_stats' that should get filled after running the functions evaluate() ,accumulate() and summarize() but 'category_stats' does not actually appear in cocoeval.py.
Is there a different version of pycocotools with this capability implemented? Or did someone plan to implement it but didn't do it yet?

[zhangmengya]

I meet the same error when i run model_main.py. The reason is self.category_stats do not exist.

Anyone could solve the problem or how to add the self.category_stats?

Thanks every much

[liuchang8am]

any updates on this one?

[ldalzovo]

Will it be fixed in the next release? Thank you

[liuchang8am]

UPDATE:
I implemented per_category_ap by extracting values from matrix s in pycocotools/cocoeval.py. It seems correct, as the mean of each category equals to the mAP calculated by the original code, so are the values from IoU 0.5:0.95:0.05. However, small, medium and large ap seems incorrect.

You may want to look at the following functions if you want to implement you own version.

1. _summerize() in pycocotools/cocoeval.py. Basically, you want to create a list to store the aps of each category, which are extracted from the 5-dimension matrix s.
2. ComputeMetrics() in metrics/coco_tools.py. Here you want to get the values stored in the list in 1), and print them with a loop over categories.

[netanel-s]

Hey @liuchang8am , thanks for the idea.
Can you please share why do you think S/M/L AP seems incorrect by this computation?
How is your implementation of per-category different than of all categories?
If it is not, then it sounds like it's either correct or that the implementation for all categories is incorrect.
Thanks again.

[pialin]

Inspired by @liuchang8am, I implemented "include_metrics_per_category"option in "eval_config", which has almost been completed by the author actually. Three code additions are listed below:
1.addtion in "tensorflow/models/research/pycocotools/cocoeval.py":from line 499 to line 648
2.addtion in "tensorflow/models/research/object_detetcion/metrics/coco_tools.py":from line 240 to line 244
3.add

{
    metrics_set: "coco_detection_metrics" 
    include_metrics_per_category: true 
}

to "eval_config" part in pipeline config file.

The code changes are only tested in my code to train a "fastrcnn_resnet_101" model. You should see per category metrics stats plotted in tensorboard if they work. You can try it if you need this feature and don't want to wait for an official implementation. Any feedback would be appreciated.

[tsoi2]

@pialin
Thanks for sharing. But your 2nd link to the coco_tools.py directs to the cocoeval.py script listed in the 1st bullet point. Could you confirm