Testing Error in TensorFlow Speech Recognition Challenge

[vvvvvjdy]

As shown in above two pic, the requirement of this competition （TensorFlow Speech Recognition Challenge）is to predict 12 classes. Those not belonging to the top 11 are all classified as 'unknown'. However, from the evaluation of mle-bench, it can be seen that the GT during the test was not mapped, resulting in very low and abnormal test results. Here, either the private test file needs to be modified or category mapping needs to be done. May I ask if this is a problem with mle-bench or kaggle? How should it be corrected?

[thesofakillers]

You are correct. This is an issue with MLE-bench, not with Kaggle.

We will get back to you about this.

[thesofakillers]

Hi, we have released an update to the readme cataloguing this issue in #66 .

TLDR:

• we won't fix this now to avoid invalidating the leaderboard, you should ignore the issue and treat this comp as usual.
• we will fix this issue, batched with other fixes when porting MLE-bench to openai/preparedness, timelines TBD.

Thanks again for flagging! Apologies for not immediately fixing.