[FT] `lighteval file` eval backend to work with stored JSONL/CSV files

[jamesbraza]

Issue encountered

I have stored LLM completions that I can export to a JSONL or CSV file, and would like to posthumously evaluate them on a few metrics.

Solution/Feature

Something like lighteval file to evaluate completions stored locally.

Possible alternatives

N/a

[NathanHB]

hi! We have something similar already, if you check for example main_vllm, you have a load_response_from_details_dat_if option. this uses details already generated by lighteval but it can be a base for your usecase.