Adds More Generative tasks #694
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| suite=["lighteval"], | ||
| prompt_function=prompt.coqa, | ||
| hf_repo="coqa", | ||
| coqa_first_question = LightevalTaskConfig( |
There was a problem hiding this comment.
For all evals in this file, either don't remove the original version (so we keep lighteval + yours) or keep the same name and change the version to 1.
There was a problem hiding this comment.
The original version didn't make any sense; each consecutive question can depend on each other. So I think this should be really the "first"version. Same for drop, it was outright broken iirc
There was a problem hiding this comment.
This is a matter of reproducibility/continuity in the eval suite: if people search for these tasks in the future they will have completely disappeared - if you just say your version is v1, it makes it easy for people to update and see what to replace things by
There was a problem hiding this comment.
Fixed it. Can I merge @clefourrier ?
There was a problem hiding this comment.
I must be blind but I still see the name as coqa_first_question not the original coqa_lighteval
There was a problem hiding this comment.
maybe I'm not looking at the correct commit though, so if this is fixed feel free to merge
There was a problem hiding this comment.
That's just variable name....
There was a problem hiding this comment.
yes XD minimal changes in PRs to avoid confusing the users, especially on evals they might be using
* add smolm generative tasks * add jeopardy * pretty 🥰 * consistent stop sequences * add versions + change names --------- Co-authored-by: Hynek Kydlicek <kydlicek.hynek@huggingface.co> Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
This PR adds more generative tasks and fixes some old ones: