Is there any way to calculate log-average miss rate?

[dungmn]

Hello,
I want to evaluate my prediction on mine dataset with YOLOv3 which is using the log-average miss rate metric?
Could you guide me how to do it?
Thanks.

[Cartucho]

Hello,
Currently, this repo is not calculating the log-average miss rate.
I will add it to my TODO list.

[dungmn]

Thanks, @Cartucho. I have another issue
I used the repo: https://github.com/AlexeyAB/darknet to train my custom objects.
When I stop training, I run ./darknet detector map data/obj.data yolo-obj.cfg backup\yolo-obj_9000.weights and get mAP around 40%.
But when I run ./darknetdetector test data/obj.data yolo-obj.cfg backup\yolo-obj_9000.weights -ext_output <data/test.txt> result.txt, I use your repo to evaluate on result.txt. However I get the mAP around 32%.
Do you know why I get 2 results differently?

[Cartucho]

Hum, that's strange indeed. Although, I am not really aware of how AlexeyAB is calculating the mAP.
In the past, PASCAL VOC used to calculate the mAP using exactly 11 recall levels however, they removed this restriction in the latest competitions.

In this repo, I use the most recent version of the official PASCAL VOC code.

[zhanghengdev]

Great work! Waiting for the log-average miss rate update. @Cartucho

[Cartucho]

Ok, I will start it now

[Cartucho]

@nguyendu392 @ZHANGHeng19931123 do you have any legit documentation explaining how to calculate the log-average miss rate?

[Cartucho]

References:

1. C. D. Manning, P. Raghavan, and H. Schutze. An Introduction to Information Retrieval. Cambridge University Press, 2008.
2. D. Hoiem, Y. Chodpathumwan, and Q. Dai. Diagnosing error in object detectors. In Proc. ECCV, 2012.
3. Dollar, Piotr, et al. "Pedestrian Detection: An Evaluation of the State of the Art." Pattern Analysis and Machine Intelligence, IEEE Transactions on 34.4 (2012): 743 - 761.

[Cartucho]

Log-average miss rate =

,

when a1, a2, ..., a9 are positive values corresponding to the miss rates at 9 evenly spaced FPPI points in log-space, between 10^-2 and 10^0.

[Cartucho]

@ZHANGHeng19931123 @nguyendu392 Done! You will now find a plot in the results/ with the log-average miss rate for each class.

[zhanghengdev]

@ZHANGHeng19931123 @nguyendu392 Done! You will now find a plot in the results/ with the log-average miss rate for each class.

Great work! Thanks a lot! While I might have spotted two small bugs:

Line 651: you calculate your lamr after map, but in your function 'voc_ap', you've changed the variable 'rec' and 'prec' by insert values at the beginning and end of your two lists.

Line 88: miss rate is 1-recall instead of 1-precision. ref: FNR=1-TPR.

And I have a small question please: I've checked the version Matlab (https://github.com/pdollar/toolbox/blob/master/detector/acfTest.m), and you have done the same things by calculating mr with miss=exp(mean(log(max(1e-10,1-miss)))). But in the paper, It is "averaging miss rate at nine FPPI rates evenly spaced in log-space". Why not use the average?

Waiting for your answer :)

[Cartucho]

Hello, you are right! I have fixed them both!

And I have a small question please: I've checked the version Matlab (https://github.com/pdollar/toolbox/blob/master/detector/acfTest.m), and you have done the same things by calculating mr with miss=exp(mean(log(max(1e-10,1-miss)))). But in the paper, It is "averaging miss rate at nine FPPI rates evenly spaced in log-space". Why not use the average?

Because it is an average in log-space: the log-average formulation of the geometric mean..
Btw, that matlab code was made by the same author as the paper.

[zhanghengdev]

Hello, you are right! I have fixed them both!

And I have a small question please: I've checked the version Matlab (https://github.com/pdollar/toolbox/blob/master/detector/acfTest.m), and you have done the same things by calculating mr with miss=exp(mean(log(max(1e-10,1-miss)))). But in the paper, It is "averaging miss rate at nine FPPI rates evenly spaced in log-space". Why not use the average?

Because it is an average in log-space: the log-average formulation of the geometric mean..
Btw, that matlab code was made by the same author as the paper.

Thank you for your reply! Have you ever checked if your result is identical with the MATLAB function 'evaluateDetectionMissRate'? I've quickly built two small scripts for the format convert and Matlab evaluation. I don't have Matlab installed, could you verify if they gave the same result?

'convert_for_matlab.py' to convert .txt files in 'ground-truth' and 'predicted' to .mat.
'lamr.m' read these .mat files to calculate miss rate with the Matlab 'evaluateDetectionMissRate' function. (before running, change work dir to 'matlab/')

May exits bugs, cause I can't debug without Matlab..
Rename these scripts before running plz, it seems that I can't upload .py and .m files :(
Thank you!

convert_for_matlab.py.txt
lamr.m.txt

[Cartucho]

@ZHANGHeng19931123 sure, thank you for those scripts, I will check it out!

I can only work on this next week because this one I'm very busy!

[niuniu111]

have anybody plot MR-FPPI curse succeedly? I use plt.plot(fppi, mr,), but the result is wrong,whether I use wrong code?looking forward for your answer