Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
53 Open 60 Closed
53 Open 60 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[FT] Add tests for nanotron feature/enhancement New feature/request good first issue Good for newcomers help wanted Extra attention is needed ignore-for-release will be ignored when doing release note
#765 opened May 21, 2025 by NathanHB
Add Romanian literals feature/enhancement New feature/request
#764 opened May 21, 2025 by dianaonutu Loading…
3
[BUG] incorrect type hints such as callable bug Something isn't working documentation Improvements or additions to documentation feature/enhancement New feature/request
#759 opened May 21, 2025 by jamesbraza
[FT] lighteval file eval backend to work with stored JSONL/CSV files feature/enhancement New feature/request
#750 opened May 20, 2025 by jamesbraza
1
[FT] add py.typed so lighteval can work with type checkers documentation Improvements or additions to documentation feature/enhancement New feature/request good first issue Good for newcomers help wanted Extra attention is needed
#749 opened May 20, 2025 by jamesbraza
[FT] allow httpx>0.27 feature/enhancement New feature/request ignore-for-release will be ignored when doing release note
#747 opened May 19, 2025 by jamesbraza
1
1
[FT] Manage script and language in the Language enum feature/enhancement New feature/request good first issue Good for newcomers
#745 opened May 19, 2025 by clefourrier
[FT] Load entire benchmark (data + spec) from the hub feature/enhancement New feature/request
#735 opened May 16, 2025 by NathanHB
@NathanHB
[FT] Add tests for VLLMModel base methods feature/enhancement New feature/request good first issue Good for newcomers
#724 opened May 15, 2025 by NathanHB
@NathanHB
[FT] Continuous batching for transformers feature/enhancement New feature/request
#723 opened May 15, 2025 by NathanHB
1
@NathanHB @ArthurZucker
[FT] Support evaluations with tool use feature/enhancement New feature/request
#719 opened May 15, 2025 by ercbot
1
Add way to load local datafile feature/enhancement New feature/request
#687 opened Apr 24, 2025 by ScottHoang Loading…
1
2
[FT] Improve Documentation and Examples documentation Improvements or additions to documentation feature/enhancement New feature/request good first issue Good for newcomers help wanted Extra attention is needed
#682 opened Apr 21, 2025 by ashim-mahara
6
Loading local data for custom tasks feature/enhancement New feature/request
#681 opened Apr 21, 2025 by ScottHoang
1
@ScottHoang
2
[FT] Log progress bar on main process feature/enhancement New feature/request help wanted Extra attention is needed
#662 opened Apr 7, 2025 by lewtun
Nanotron model updates feature/enhancement New feature/request
#652 opened Mar 31, 2025 by anton-l Loading…
1
[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split feature/enhancement New feature/request good first issue Good for newcomers help wanted Extra attention is needed
#634 opened Mar 21, 2025 by EdwardSJ151
added vllm lora support. feature/enhancement New feature/request
#611 opened Mar 11, 2025 by haizhou-shi Loading…
4
[FT] Assistant Response Prefilling feature/enhancement New feature/request
#591 opened Feb 28, 2025 by matmult
[FT] LiteLLM concurrency parameters hard-coded feature/enhancement New feature/request good first issue Good for newcomers help wanted Extra attention is needed
#567 opened Feb 16, 2025 by lhl
<think> tags for thinking models feature/enhancement New feature/request
#513 opened Jan 24, 2025 by JoelNiklaus
1
2
[FT] Enable lazy model initialization feature/enhancement New feature/request
#496 opened Jan 11, 2025 by JoelNiklaus
[FT] Custom model to TransformersModel feature/enhancement New feature/request
#489 opened Jan 7, 2025 by Giuseppe5
14
[FT] Add and test multinode runs back feature/enhancement New feature/request
#482 opened Jan 2, 2025 by clefourrier
[FT] Enhancing CorpusLevelTranslationMetric with Asian Language Support feature/enhancement New feature/request
#478 opened Dec 27, 2024 by ryan-minato
ProTip! What’s not been updated in a month: updated:<2025-05-03.