Search results

[BUG] System Prompt option now missing?

bugSomething isn't workingSomething isn't working

Status: Open.

#844 In huggingface/lighteval;

· teknium1 opened on Jun 27, 2025

[BUG] Failed downloads / 429 rate limiting for concurrent evaluations sharing cache

bugSomething isn't workingSomething isn't working

Status: Open.

#813 In huggingface/lighteval;

· lukemerrick opened on Jun 16, 2025

[BUG] <code>Registry.task_registry</code> not reimporting the custom task module

bugSomething isn't workingSomething isn't working

Status: Open.

#805 In huggingface/lighteval;

· jamesbraza opened on Jun 10, 2025

[BUG] Unable to Download Dataset in Restricted Network Environment (No Access to Hugging Face or Mirrors)

bugSomething isn't workingSomething isn't working

Status: Open.

#802 In huggingface/lighteval;

· xjli360 opened on Jun 9, 2025

[BUG] <code>test</code> split forced to hit <code>Careful</code> warning

bugSomething isn't workingSomething isn't working

Status: Open.

#801 In huggingface/lighteval;

· jamesbraza opened on Jun 7, 2025

[BUG] Forced to hit <code>You cannot select the number of dataset splits</code> with <code>litellm</code>

bugSomething isn't workingSomething isn't working

Status: Open.

#800 In huggingface/lighteval;

· jamesbraza opened on Jun 7, 2025

[BUG] support direct evaluation of local API and local data sets?

bugSomething isn't workingSomething isn't working

Status: Open.

#793 In huggingface/lighteval;

· HemiFate opened on Jun 5, 2025

[BUG] Is AIME24 broken?

bugSomething isn't workingSomething isn't working

Status: Open.

#771 In huggingface/lighteval;

· eldarkurtic opened on May 22, 2025

[FT] Python API docs using small model that can run on Mac

bugSomething isn't workingSomething isn't working

documentationImprovements or additions to documentationImprovements or additions to documentation

Status: Open.

Bug

#761 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[BUG] custom model docs don't run: missing imports

bugSomething isn't workingSomething isn't working

documentationImprovements or additions to documentationImprovements or additions to documentation

good first issueGood for newcomersGood for newcomers

Status: Open.

Bug

#760 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[BUG] incorrect type hints such as <code>callable</code>

bugSomething isn't workingSomething isn't working

documentationImprovements or additions to documentationImprovements or additions to documentation

feature/enhancementNew feature/requestNew feature/request

Status: Open.

Bug

#759 In huggingface/lighteval;

· jamesbraza opened on May 21, 2025

[BUG] Make Probability metric not trigger Loglikehood requests twice when used alongside Accuracy Metrics

bugSomething isn't workingSomething isn't working

science-team

Status: Open.

Bug

#751 In huggingface/lighteval;

· hynky1999 opened on May 20, 2025