nan

[insafim]

There are some question with "nan" in the options. During evaluations are the "nan" choices sent too to the model?

For example:

944	Think about the magnetic force between the magnets in each pair. Which of the following statements is true? Options: A: The magnitude of the magnetic force is smaller in Pair 1., B: The magnitude of the magnetic force is the same in both pairs., C: The magnitude of the magnetic force is smaller in Pair 2., D: nan

945	Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end. Question: Is the number of big objects that are in front of the metal fighter less than the number of things that are behind the big metallic bus? Choices: (A) Yes (B) No (C) nan (D) nan

[insafim]

@kennymckormick @xiaoachen98 @Li-Jinsong

[kennymckormick]

Hi, @insafim ,
MMStar officially adopts VLMEvalKit as the evaluation toolkit. In VLMEvalKit implementation, empty choices will not be feeded to the VLM.

[insafim]

391	How many angels have halos? Options: A: Two, B: One, C: nan, D: Three		D	fine-grained perception	object counting

268	How many tree stumps are there in the image? Options: A: nan, B: Two, C: One, D: Cannot be determined		A	fine-grained perception	object counting

But if you take these questions not feeding nan will alter the answer?

How do deal with these cases

[sunlaito]

I also found the 'nan' problem. In addition, simply changing "nan" to "None" cannot solve all problems.
Some questions (most from MMBench) only have 2 valid options, A and B. The options C and D of these questions were the empty strings "" and need not be added to the query to LLM.

Some questions from other bench dataset, the "nan" options should be "None" instead. This option usually appears in a question of counting some objects in an image.