Refcoco/+/g problem

[tzjtatata]

Hi. I am trying to reproduce your results and compare with other REC/Detection works. But I find that the Refcoco/+/g benchmark in your paper seems different with the official version. While Refcoco validation has 10834 expressions and the expressions reported in Table 1 is only 5786.

Why do you delete some expressions in the benchmark? Are there any reasons?

[LiuRicky]

Thanks for your interest.

Actually there are only 5786 instances (1w+ expressions but only 5786 instances).

For evaluation efficiency, we randomly pick 1 expression for each instance. Actually our evaluation will result in a relatively lower score. You can try more expression and I guess the metrics will be higher than we reported.

[tzjtatata]

Thank you for reply. But this modification of the benchmark may make this work hard to comparison with others.

[tzjtatata]

By the way, I notice that you use entropy loss and kl loss while GRPO do not use entropy loss and this paper do not mention it. I wonder whether there is an ablation for this modification?

[LiuRicky]

Thank you for reply. But this modification of the benchmark may make this work hard to comparison with others.

Our evaluation for MLLMs are under the same evaluation protocol and using the same code base. The results are comparable so long as the evaluation is under our codes and framework.

[LiuRicky]

By the way, I notice that you use entropy loss and kl loss while GRPO do not use entropy loss and this paper do not mention it. I wonder whether there is an ablation for this modification?

Thanks for your point.

We follow the training protocol of EasyR1. I would be grateful if you can ablate this modification and provide us some insights.

[LiuRicky]

Further verification and explanation

Since we count the instance level, the 5786 is 3811(val) + 1975(test). We write in the table caption that "We report the number of instances for detection and segmentation tasks." and evaluations are done under the same protocol.

[tzjtatata]

i have tested. The performance is similar. But the full evaluation should be better for comparison.

…

---Original--- From: ***@***.***> Date: Mon, Jun 23, 2025 23:37 PM To: ***@***.***>; Cc: "Yuanze ***@***.******@***.***>; Subject: Re: [dvlab-research/VisionReasoner] Refcoco/+/g problem (Issue #12) LiuRicky left a comment (dvlab-research/VisionReasoner#12) Further verification and explanation Since we count the instance level, the 5786 is 3811(val) + 1975(test). We write in the table caption that "We report the number of instances for detection and segmentation tasks." and evaluations are done under the same protocol. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[LiuRicky]

We have just tested and the results are better. There are around ~1% performance gain. For reference:

RefCOCO-val	RefCOCO-testA	RefCOCO+-val	RefCOCO+-testA	RefCOCOg-val	RefCOCOg-test
89.1	91.0	85.0	87.6	87.6	88.5

[tzjtatata]

Thank you for reply. Excellent.