How to evaluate on a smaller split of the data?

[armsp]

Thank you so very much for this project!
Currently I am using your framework to evaluate some models I have trained, however for me all 26529 samples are not necessary to run the evaluation. Moreover I have not yet figured out how to run it faster since it takes up quite a bit of time.

I understand that there is a --split parameter that seems to take the value SuperGPQA-all. I filtered the dataset just to have the fields I am interested in and made a new data file and named it as supergpqa-mmp.jsonl and I passed the argument to --split as supergpqa-mmp, but that did not work.

Could you please tell me how do I run the evaluation/inference on just a subset of fields or disciplines that I am interested in?

[mkj3085003]

Hi! Thanks a lot for your message — really glad to hear you're using the framework!

You're absolutely right to try customizing the evaluation by using your own filtered subset (e.g., supergpqa-mmp.jsonl). The reason it's not working currently is that the load_data function in infer/data_loader.py has a hardcoded condition that only allows the "SuperGPQA-all" split:

def load_data(split='', mode=''):
    if split in ["SuperGPQA-all"] and mode in ['zero-shot', 'zero-shot-bon', 'five-shot']:
        ...

This means any custom split like supergpqa-mmp will be ignored.

✅ Suggested Fix

To allow custom splits, you can modify the condition as follows:

def load_data(split='', mode=''):
    if mode in ['zero-shot', 'zero-shot-bon', 'five-shot']:
        ...

This way, the loader won’t restrict the split name and will accept your custom file (e.g., supergpqa-mmp.jsonl) as long as it’s placed in the expected location, like under the data/ directory.