1 Release published by 1 person

• v0.10.0

  published May 22, 2025

16 Pull requests merged by 6 people

• Async vllm

  #693 merged May 22, 2025

• Bump ruff version

  #774 merged May 22, 2025

• Nanotron, Multilingual tasks update + misc

  #756 merged May 22, 2025

• Add missing model_name fixes

  #768 merged May 21, 2025

• add dependencies to run after pip install

  #767 merged May 21, 2025

• fix custom model example

  #766 merged May 21, 2025

• Adds template for custom path saving results

  #755 merged May 21, 2025

• Allow for model kwargs when loading transformers from pretrained

  #754 merged May 21, 2025

• Add MCQ support to Yourbench evaluation

  #734 merged May 20, 2025

• Fix task metric type mismatch

  #743 merged May 20, 2025

• Adds multimodal support and MMMU pro

  #675 merged May 19, 2025

• Fix extractive match

  #746 merged May 19, 2025

• Added Flores

  #717 merged May 19, 2025

• Update main_endpoint.py

  #739 merged May 19, 2025

• Fix litellm

  #736 merged May 16, 2025

• Adds More Generative tasks

  #694 merged May 16, 2025

9 Pull requests opened by 7 people

• Add Chinese (zh) Translation of Documentation

  #744 opened May 19, 2025

• Newer `openai` and loosened `httpx`

  #758 opened May 21, 2025

• Add Romanian literals

  #764 opened May 21, 2025

• Add Bulgarian and Macedonian literals

  #769 opened May 22, 2025

• Add TranslationLiterals for Language.DANISH

  #770 opened May 22, 2025

• Add support for vLLM KV-cache quantization

  #773 opened May 22, 2025

• Update translation_literals.py with icelandic

  #775 opened May 22, 2025

• Bump dev version to 0.10.1.dev0

  #777 opened May 22, 2025

• Complete TranslationLiterals for Language.ESTONIAN

  #779 opened May 23, 2025

10 Issues closed by 3 people

• [FT] bump ruff version

  #772 closed May 22, 2025

• [BUG] fix dependencies when doing fresh install

  #725 closed May 21, 2025

• [BUG] pydantic throws error with custom evaluator

  #757 closed May 21, 2025

• [FT] Custom details and results saving path

  #753 closed May 21, 2025

• [FT] better support for model loading args in transformers

  #752 closed May 21, 2025

• [BUG] Python API docs generating splits forever

  #762 closed May 21, 2025

• [FT] Add multimodal for transformers models

  #729 closed May 19, 2025

• [EVAL] adds FLORES

  #727 closed May 19, 2025

• [BUG] remove use chat template flag for litellm

  #738 closed May 19, 2025

• [FT] Controlling the number of experiments/trials to run

  #718 closed May 16, 2025

17 Issues opened by 7 people

• [EVAL] GSM Plus

  #778 opened May 23, 2025

• [BUG] Is AIME24 broken?

  #771 opened May 22, 2025

• [FT] Add tests for nanotron

  #765 opened May 21, 2025

• [FT] Support computing/displaying stddev in the final table from the variation in sample scores (when using sampling metrics) instead of a bootstrap

  #763 opened May 21, 2025

• [FT] Python API docs using small model that can run on Mac

  #761 opened May 21, 2025

• [BUG] custom model docs don't run: missing imports

  #760 opened May 21, 2025

• [BUG] incorrect type hints such as `callable`

  #759 opened May 21, 2025

• [BUG] Make Probability metric not trigger Loglikehood requests twice when used alongside Accuracy Metrics

  #751 opened May 20, 2025

• [FT] `lighteval file` eval backend to work with stored JSONL/CSV files

  #750 opened May 20, 2025

• [FT] add `py.typed` so `lighteval` can work with type checkers

  #749 opened May 20, 2025

• [BUG] sync `LightevalTaskConfig` docstring with types/defaults

  #748 opened May 20, 2025

• [FT] allow `httpx>0.27`

  #747 opened May 19, 2025

• [FT] Manage script and language in the Language enum

  #745 opened May 19, 2025

• [BUG] Sampling and max new tokens params for accelerate backend not being applied correctly

  #742 opened May 19, 2025

• [EVAL] TauBench:

  #741 opened May 19, 2025

• [EVAL] SciCode: reasearch coding benchmark

  #740 opened May 19, 2025

• Error with value of `n`

  #737 opened May 19, 2025

11 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Adds RULER benchmark

  #722 commented on May 21, 2025 • 10 new comments

• [EVAL] Adding PHARE

  #696 commented on May 18, 2025 • 0 new comments

• [FT] Custom model to TransformersModel

  #489 commented on May 19, 2025 • 0 new comments

• [BUG] Installing lighteval breaks hydra-core

  #713 commented on May 19, 2025 • 0 new comments

• Call for contributions: Translate lighteval's doc into Chinese

  #716 commented on May 19, 2025 • 0 new comments

• [FT] Add tests for `VLLMModel` base methods

  #724 commented on May 20, 2025 • 0 new comments

• [FT] Continuous batching for transformers

  #723 commented on May 20, 2025 • 0 new comments

• Loading local data for custom tasks

  #681 commented on May 20, 2025 • 0 new comments

• Making bootstrap_iters an arg

  #697 commented on May 21, 2025 • 0 new comments

• [WIP] Fix nanotron compatibility

  #706 commented on May 20, 2025 • 0 new comments

• update for CB

  #714 commented on May 19, 2025 • 0 new comments