Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
118 Open 1,487 Closed
118 Open 1,487 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

[Add Dataset Update] KBL 2025
#3000 opened May 20, 2025 by abzb1 Loading…
Enable text-only evals for VLM models
#2999 opened May 19, 2025 by ysulsky Loading…
1
add Mbpp instruct
#2995 opened May 19, 2025 by baberabb Loading…
1
Output path fix
#2993 opened May 19, 2025 by Niccolo-Ajroldi Loading…
1
1
[Fix] Update resolve_hf_chat_template arguments
#2992 opened May 19, 2025 by fxmarty-amd Loading…
1
Fix error due in Collating queries with different continuation lengths (fixes #2984)
#2987 opened May 15, 2025 by ameyagodbole Loading…
1
3
[longbench] fix metric calculation
#2983 opened May 14, 2025 by baberabb Loading…
use images with api models
#2981 opened May 14, 2025 by baberabb Loading…
1
Add mlqa tag to run all variants.
#2977 opened May 12, 2025 by ivanbaldo Loading…
6
add image hashing
#2973 opened May 11, 2025 by artemorloff Loading…
3
Adding resize images support
#2958 opened May 6, 2025 by artemorloff Loading…
add ultravox models support for audio tasks
#2957 opened May 6, 2025 by artemorloff Loading…
Added RuBLiMP, a Russian benchmark of linguistic minimal pairs
#2951 opened May 2, 2025 by vmkhlv Loading…
2
Final putnam axiom bm
#2946 opened May 1, 2025 by brando90 Loading…
4
feat: Add LIBRA benchmark for long-context evaluation
#2943 opened Apr 30, 2025 by karimovaSvetlana Loading…
4 tasks done
1
Resolve the inconsistency between the description of output_path and the actual logic
#2928 opened Apr 24, 2025 by rangehow Loading…
1
Fix gsm8k task to enhance accuracy
#2924 opened Apr 21, 2025 by hfadzxy Loading…
5
Added selection filter: take_last
#2923 opened Apr 18, 2025 by JamesClarke7283 Loading…
3 tasks done
2
enable few-shots for multimodal tasks
#2912 opened Apr 15, 2025 by artemorloff Loading…
1
Adapting multimodal tasks for phi3.5 vision
#2909 opened Apr 14, 2025 by artemorloff Loading…
2
Audio modality: add openbmb/MiniCPM-o-2_6 model
#2908 opened Apr 14, 2025 by artemorloff Loading…
1
Add Qwen/Qwen-Audio-Chat support for audio modality
#2906 opened Apr 14, 2025 by artemorloff Loading…
2
enable evaluation from yaml config file
#2893 opened Apr 8, 2025 by artemorloff Loading…
7
Added AIME Support
#2892 opened Apr 8, 2025 by Zephyr271828 Loading…
2 of 3 tasks
5
Fix GPQA CoT n shot
#2888 opened Apr 7, 2025 by anmarques Loading…
4
ProTip! Updated in the last three days: updated:>2025-05-17.